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ABSTRACT 


This  thesis  reviewed  two  different  approaches  to  benefit  analysis, 
benefit  comparison  and  user  satisfaction,  that  could  be  applied  to  the 
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the  GSA  specifies  that  non-cost  factors  should  be  evaluated  in  such  an 
analysis. 

Different  benefit  comparison  and  user  satisfaction  methods  are 
reviewed  for  their  particular  advantages  and  disadvantages.  A 
discussion  is  given  on  how  selected  methods  of  each  approach  may  be 
us3d  within  the  federal  government  for  the  evaluation  of  alternatives. 
Suggestions  are  made  for  ways  of  conducting  a  more  complete  analysis 
of  alternatives  through  incorporating  present  value  analysis,  benefit 
comparison  of  non-cost  factors,  and  analysis  of  user  satisfaction,  into 
one  comprehensive  analysis. 
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I .  INTRODUCTION 


A.  BACKGROUND 

The  need  for  benefit  analysis  methods  capable  of  being 
applied  to  the  evaluation  of  proposed  information  systems 
has  long  been  recognized.  Twenty  years  ago  Chervany  and 
Dickson  (Chervany  and  Dickson, 1970)  asserted  then  that  it 
was  time  to  re-evaluate  the  decision-making  process 
involving  the  development  of  information  system 
alternatives.  In  the  mean  time  the  demand  for  information 
systems,  and  hence  the  demand  for  benefit  analysis  of  these 
systems,  has  grown  substantially.  Abdel -Hamid  and  Madnick 
(Abdel-Hamid  and  Madnick, 1989)  estimate  the  increase  in 
demand  at  a  hundredfold  over  the  last  two  decades.  Kitfield 
(Kitfield,  1989)  estimated  the  demand  for  avionic  software 
was  increasing  25  percent  per  year.  This  trend  may  be  even 
more  true  for  the  Department  of  Defense  (DoD)  as  one  expert 
has  described  what  he  calls  the  "military  software  crisis" 
(Kitfield, 1989) .  Kitfield  describes  this  crisis  as, 

"runaway  demand  and  a  profound  shortage  of  software 
programmers"  (Kitfield, 1989) .  The  problem  of  selecting  an 
information  system  alternative  may  now  be  focused  more  on 
which  alternative  to  invest  in  first.  This  situation 
reinforces  the  need  for  benefit  analysis  to  prioritize 
software  projects. 
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B.  OBJECTIVES 


The  objective  of  this  thesis,  is  to  determine  what 
methodologies  can  be  used  to  analyze  the  benefits  to  be 
gained  from  a  proposed  information  system.  More 
specifically  the  objective  is  to  examine  what  current 
benefit  analysis  methods  can  be  used  in  the  evaluation  of 
different  competitive  bids  submitted  in  response  to  a 
solicitation  from  the  federal  goveriunent  for  a  proposed 
information  system.  Of  special  interest  is  how  intangible 
benefits  of  advanced  information  technology  are  treated  in 
such  an  analysis. 

C.  RESEARCH  QUESTIONS 

The  research  concerns  of  this  thesis  are  really  two 
fold.  First,  which  benefit  analysis  methods  can  be  employed 
to  compare  the  benefits  of  two  competing  proposed 
information  systems  to  determine  the  alternative  that  is  the 
most  advantageous  to  the  government.  Secondly,  which 
benefit  analysis  methods  are  capable  of  calculating  a 
monetary  value  of  the  benefits  from  a  complex  information 
system  prior  to  that  system's  development. 

D.  SCOPE  AND  LIMITATIONS 

This  study  will  focus  on  benefit  analysis  methods  that 
can  be  used  in  the  evaluation  of  alternatives  which  is 
conducted  in  the  process  of  a  federal  acquisition  of  an 
information  system.  Benefit  analysis  methods  will  also  be 
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examined  that  can  be  incorporated  into  a  larger  cost-benefit 
analysis.  The  subject  of  risk  assessment,  in  relation  to 
information  systems,  is  included  under  benefit  analysis  of 
proposed  information  systems  for  the  purpose  of  this  thesis. 
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I I .  METHODOLOGY 


A.  SELECTING  STUDIES  OP  BENEFIT  ANALYSIS 

1.  Choosing  a  Medium  from  which  to  Select  Studies 
Published  articles  were  chosen  as  the  source  of 

studies  on  benefit  analysis  in  both  existing  and  proposed 
information  systems.  Studi  s  on  existing  systems  were 
included  to  gain  a  better  insight  into  the  subject  of 
benefit  analysis  of  proposed  information  systems.  Only 
articles  of  benefit  analysis  in  English-language  journals, 
in  the  field  of  management  information  systems  and  the 
related  field  of  management  science,  were  surveyed. 

2.  Choosing  a  Time  Span  of  the  Studies  to  Survey 
The  work  on  this  literature  review  began  in  June, 

1990  and  was  completed  in  January,  1991.  The  last  articles 
that  were  excepted  for  inclusion  were  published  in  the 
calendar  year  of  1990.  The  year  1970  was  selected  as  the 
starting  point  for  the  bibliographical  search  of  this 
literature  review.  However,  a  time  limit  was  not  placed  on 
articles  or  supporting  publications  that  could  be 
incorporated  into  the  research  based  on  references  in 
articles  found  in  the  main  bibliographical  search  or  books 
that  supported  the  underlying  concepts  of  an  article. 
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3 .  Search  for  Key  Words 

a.  The  Initial  Bibliographical  Search 

An  initial  search  was  conducted  manually  of 
Harvard  Business  Review  and  Management  Information  Systems 
Quarterly  to  find  key  words  that  could  be  used  in  the  search 
of  on-line  bibliographic  data.  These  journals  were  used 
because  of  their  history  of  articles  published  in  the  field 
of  management  information  systems.  The  table  of  contents  of 
every  issue  between  1970  and  1990  was  manually  reviewed.  If 
a  title  appeared  relevant  to  the  subject  of  benefit  analysis 
of  information  systems  then  the  article  was  reviewed.  After 
reading  the  article  and  determining  that  indeed  it  did 
pertain  to  the  subject,  the  article  was  then  copied  and 
saved  for  later  referral. 

b.  Key  Words  Found 

The  search  for  key  words  identified  19  articles 
from  these  two  journals.  From  the  initial  search  it  could 
also  be  concluded  that  the  authors  of  the  19  articles  chose 
diverse  key  words  to  describe  their  articles.  Key  words 
included: 

-  Cost/Benefit  Analysis 

-  Cost/Benefit  Calculations 

-  Information  Economics 

-  Information  System  Investment 

-  Information  Technology  Investment 

-  Value  Analysis 
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As  the  dissimilarity  of  key  words  found  in  the  initial 
search  was  discovered,  it  was  determined  that  a  single  on~ 
line  search  with  a  single  set  of  key  words  would  not 
sufficiently  identify  the  number  of  articles  desired  on  the 
subject  of  benefit  analysis  of  information  systems.  The 
problem  was  that  the  key  words  that  were  found  were  just  to 
diverse  for  the  on-line  system  to  be  effective. 

4.  Manual  Bibliographical  Search 

As  a  result  of  the  difficulties  with  key  words,  a 
manual  bibliographical  search  was  undertaken  as  opposed  to 
an  on-line  search.  This  main  search  was  conducted  in  the 
same  manner  as  the  preliminary  search.  A  list  of  eight 
additional  journals  was  gathered  based  on  their  reputation 
for  publishing  studies  in  the  field  of  management 
information  systems.  The  journals  searched  manually  were: 

-  Communications  of  the  ACM  * 

-  Data  Base 

-  Decision  Sciences 

-  Harvard  Business  Review 

-  Information  and  Management 

-  Journal  of  Management  Information  Systems 

-  Management  Science 

-  Management  Information  Systems  Quarterly 

-  Omega 

-  Sloan  Management  Review 

*  ACM:  Association  for  Computing  Machinery 
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a.  Conduct  of  the  Manual  Searcb 

Once  the  list  of  journals  was  established  the 
search  was  conducted  systematically.  Journal  by  journal, 
the  table  of  contents  for  each  issue  available,  during  the 
time  period  of  1970  to  1990,  was  examined  for  a  possible 
article  relevant  to  the  subject  of  interest.  Articles  with 
fitting  titles  were  copied  for  latter  reference  if  after 
being  reviewed  it  was  concluded  that  they  were  appropriate. 

b.  Other  Sources  of  Literature 

Other  sources  of  reference  were  considered  to 
clarify  certain  points  brought  up  in  the  articles  that  had 
been  initially  collected  and  reviewed.  The  references  of 
the  articles  located  in  the  main  search  were  used  to 
identify  further  articles  that  could  pertain  to  the  subject 
of  this  study.  These  sources  were  located  through  on-line 
bibliographical  searches  by  either  author,  title,  or 
subject.  The  on-line  search  was  useful  to  locate  these 
additional  sources  within  the  library  for  in  this  instance 
very  specific:  author  names,  and  titles,  were  used  to  query 
the  on-line  system  for  exact  locations  of  individual  works. 

A  final  source  of  literature  that  comprises  the  foundation 
of  this  thesis  came  from  the  courses  that  this  author  was 
exposed  to  at  the  Naval  Postgraduate  School .  A 
comprehensive  inventory  of  the  literature  found  and  used  for 
this  thesis  is  presented  in  the  bibliography. 
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B.  SORTING  THE  WORKS  ON  BENEFIT  ANALYSIS 

In  combination,  the  preceding  search  methods  yielded  a 
total  of  50  references  that  were  considered  for  possible 
contribution  to  the  study  of  benefit  analysis  of  proposed 
information  systems.  These  references  were  organized  into 
three  groups:  benefit  comparison,  user  satisfaction,  and 
miscellaneous  literature. 

1.  Benefit  Comparison 

The  articles  in  this  group  are  distinguished  from 
the  others  by  the  condition  that  they  present  methods  of 
analyzing  benefits  by  assessing  various  system  attributes. 
The  methods  discussed  under  benefit  comparison  are:  the 
additive  weight  model,  the  eigenvector  model,  and  the  multi¬ 
attribute  utility  model.  Chapter  IV  is  dedicated  to  benefit 
comparison. 

2.  User  Satisfaction 

The  references  in  this  group,  in  contrast  to  the 
groups  previously  mentioned,  propose  to  analyze  benefits  not 
through  an  objective  indicator,  but  through  a  survey  to 
determine  the  subjective  satisfaction  that  users  have  with 
their  respective  information  system.  The  procedures  of  this 
method  incorporate  various  formulas  and  models  to  interpret 
the  responses  of  user  satisfaction  questionnaires.  User 
satisfaction  is  the  subject  of  Chapter  V. 
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3.  Miscellaneous  Literature 

The  last  group  that  references  were  organized  into 
was  miscellaneous.  This  group  contained  articles  on  the 
importance  of  information  systems  and  their  predominance 
throughout  the  business  world  and  DoD.  Articles  were  also 
placed  in  this  group  that  stressed  the  value  of  information 
and  the  need  fcvr  benefit  analysis  of  information  systems. 
Articles  on  the  competitive  advantage  to  be  gained  from 
information  systems  were  also  placed  in  the  miscellaneous 
group.  In  spite  of  the  expressed  need  for  benefit  analysis, 
the  group  of  miscellaneous  literature  did  not  contribute  to 
any  specific  benefit  analysis  technique.  As  such,  the 
references  in  this  group  were  largely  used  as  background 
material  that  stressed  the  importance  of  information  system 
benefit  analysis  and  that  put  benefit  analysis  in 
perspective  in  relation  to  analysis  of  alternatives  and  the 
rest  of  the  development  life  cycle  for  information  systems. 


9 


III.  BACKGRODMD 


h.  GUIDELINES  FOR  FEDERAL  ADP  ACQUISITIONS 

The  Brooks  Act,  public  law  89-306,  established  the  U.S 
General  Services  Administration  (GSA)  as  the  procurement 
authority  for  ADP  equipment  acquisitions  within  the  federal 
government.  The  Act  instructs  the  GSA  to  "provide  for  the 
economic  and  efficient  purchase,  lease,  and  maintenance  of 
automated  data  processing  equipment  by  federal  agencies" 
(U.S.  GSA, 1990).  This  authority  was  increased  through  the 
enactment  of  public  law  99-500  to  also  include  the  federal 
acquisition  of  such  things  as;  software,  firmware,  and 
computer  related  contracted  services  to  include  programming 
and  support  services.  In  fulfillment  of  their 
responsibility  the  GSA  publishes  both  procedural  guides, 
such  as  A  Guide  for  Requirements  Analysis  and  Analysis  of 
Alternatives .  and  the  federal  information  resources 
management  regulation  (FIRMR).  (U.S.  GSA, 1990) 

The  Warner  Amendment,  as  enacted  under  public  law  99- 
500,  did  release  DoD  from  the  jurisdiction  of  the  Brooks  Act 
for  certain  ADP  functions  that  dealt  with  national  security 
or  military  operations.  However,  the  Warner  Amendment  did 
not  release  DoD  from  the  conditions  of  the  Brooks  Act  for 
ADP  acquisitions  that  deal  with  a  vast  number  of  other 
functions  such  as  logistics  and  administrative  activities. 
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As  such  the  DoD  must  abide  by  PIRMR  when  making  a  great 
number  of  ADP  and  information  system  acquisitions. 

B.  GSA  POLICY  FOR  THE  AHALTSI8  OP  ALTERHATIVES 

In  the  process  of  making  a  federal  acquisition  of  ADP 
equipment  or  systems  the  GSA  requires  that  an  analysis  of 
alternatives  be  performed.  The  objective  of  this  analysis 
is  to  determine  the  most  advantageous  alternative  to  the 
government.  As  a  part  of  this  analysis  of  alternatives  the 
GSA  has  mandated  that  both  cost  and  non-cost  factors  are  to 
be  evaluated.  The  specific  cost  and  non-cost  factors  that 
should  be  considered  in  this  type  of  analysis  have  been 
determined  by  the  GSA  and  are  listed,  and  described,  in  A 
Guide  for  Requirements  Analysis  and  Analysis  of  Alternatives 
(U.S.  GSA, 1990).  The  cost  factors  are: 
conversion 
personnel 
supplies 
energy 

-  maintenance 
space 

administrative  costs  of  contracting 
contract  price 

The  cost  factors  above  are  all  quite  tangible  and  relatively 
easy  to  evaluate  in  the  course  of  an  analysis  of 
alternatives.  In  fact  the  GSA  has  recommended  three 
different  methods  for  use  in  the  analysis  of  cost  factors: 
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present  value  analysis,  net  present  value,  and  benefit-cost 
ratio.  (U.S.  GSA,1990) 

C.  AHALTSIS  OF  COST  FACTORS 
1.  Description 

Of  the  techniques  recommended  by  GSA  for  the 
analysis  of  cost  factors,  present  value  analysis  is  common 
to  capital  investment  and  calculates  the  present  worth  of  a 
monetary  amount  to  be  received  in  the  future.  Present  value 
analysis  takes  into  account  the  time  value  of  money,  or  the 
concept  that  a  dollar  is  worth  more  today  than  a  year  from 
today.  There  are  several  different  approaches  to  present 
value  analysis  to  included:  present  value  of  a  single  future 
payment,  present  value  net  benefits,  and  present  value  for 
competing  alternatives. 

a.  Present  Value  of  a  Single  Future  Payment 

In  Garrison  (1988),  present  value  analysis  is 
described  in  the  context  of  a  the  receipt  of  a  single 
amount,  or  benefit,  in  the  future.  The  formula  for 
calculating  present  value  of  a  one  time  payment  in  the 
future  is  given  in  Formula  1.  (Garrison, 1988) 


pm - 

(l*x)“  (1) 

In  Formula  1,  P  represents  the  present  value  of 
the  benefit  in  monetary  terms.  F  is  the  monetary  value  of 
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the  benefit  to  be  received  in  the  future,  r  is  the  interest 
rate  and  n  is  the  number  of  years  in  the  future  that  the 
benefit  will  be  received.  (Garrison, 1988) 
b.  Present  Value  Met  Benefits 

Thompson  (1980)  examines  present  value  analysis 
in  terms  of  discovering  the  sum  of  all  the  systems  results, 
both  good  and  bad,  or  in  other  words  both  liabilities  and 
assets.  In  his  work  Thompson  uses  the  expression  "present¬ 
valued  compensating  variation”  when  referring  to  the 
difference  between  the  cost  and  benefit  of  a  particular 
attribute.  Thompson's  formula  for  present-valued  net 
benefits  is  given  in  Formula  2.  (Thompson, 1980) 


n  a 


CVr 


(2) 


In  Formula  2,  CV^j  is  the  compensating  variation 
for  benefit  j  on  the  user  i.  d  is  the  discoxmt  rate,  y  is 
the  number  of  years  until  the  user  i  is  affected  by  benefit 
j.  m  is  the  number  of  different  benefits,  n  is  the  ntunber 
of  users  benefitted  by  the  system.  (Thompson, 1980) 

Thompson's  present -valued  net  benefits  method 
should  not  be  confused  with  the  net  present  value  (NPV) 
method.  As  recommended  by  GSA,  the  NPV  is  equal  to 
difference  between  the  total  present  value  of  all  of  the 
benefits  and  the  total  present  value  of  all  of  the  costs. 
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NPV,  unlike  present-valued  net  benefits,  does  not  attempt  to 
sum  the  differences  between  costs  and  benefits  of  each 
characteristic  of  an  information  system.  (U.S.  GSA,1990) 
c.  Present  Value  for  CoBv>eting  Alternatives 
In  NAVDAC  PUB  15  (Dec, 1980)  present  value 
analysis  is  used  to  evaluate  alternatives  based  on  lowest 
cost.  In  this  application  of  present  value  analysis,  the 
benefits  of  all  alternatives  being  considered  must  be  equal 
and  a  selection  is  made  by  determining  the  lowest  present 
value  of  the  costs  of  the  different  alternatives.  Also,  by 
this  method,  the  service  lives  of  the  alternatives  must  be 
finite  and  equal  among  alternatives,  or  be  placed  in  equal 
terms.  In  computing  present  values  the  NAVDAC  method  uses 
standard  tables  of  discount  factors  in  which  the  present 
value  of  a  dollar  can  be  found  for  both  a  single  cash  flow 
and  cumulative  uniform  cash  flows.  Uniform  here  being  the 
same  amount  paid  every  year.  This  form  of  the  present  value 
may  be  easier  to  conceptualized  if  placed  in  the  context  of 
hardware  acquisitions.  (NAVDAC  PUB  15,1980) 

2.  Example  of  Present  Value  Analysis 

In  this  example  two  high  capacity  printers  are  under 
consideration  for  purchase.  Both  of  the  printers  have  the 
same  exact  capabilities,  such  as  the  same  output  speeds, 
font  styles,  and  buffer  sizes.  One  of  the  printers, 
alternative  A,  has  a  service  life  of  six  years.  The  other, 
alternative  B,  has  a  service  life  of  three  years.  In  this 
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situation  the  NAVDAC  method  of  present  value  analysis  can  be 
applied  for  the  benefits  are  equal  and  the  service  lives  can 
be  placed  in  equal  terms.  (NAVDAC  PUB  15,1980) 

To  further  define  the  example,  alternative  A  costs 
$10,000  to  purchase  and  has  annual  reoccurring  cost  of 
$4,000.  Alternative  B  costs  $8,000  to  purchase  and  has 
annual  reoccurring  costs  of  $5,000.  Both  alternatives  are 
not  expected  to  retain  any  value  at  the  end  of  the  economic 
lives.  (NAVDAC  PUB  15,1980) 

a.  Present  Value  Using  a  Six  Year  Service  Life 

The  present  value  for  the  two  alternatives  using 
a  six  year  service  life  would  be  found  as  follows.  For 
alternative  A  the  purchase  price  would  be  added  to  the 
reoccurring  expenses  multiplied  by  the  factor  for  a  six  year 
cumulative  uniform  series.  In  alternative  B  two  printers 
have  to  be  purchased  so  that  their  combined  service  life 
equals  six  years.  The  present  value  of  alternative  B  is  the 
purchase  price  for  the  first  printer  added  to  the 
reoccurring  expenses  multiplied  by  the  factor  for  a  six  year 
cumulative  uniform  series  and  then  added  to  the  purchase 
price  of  the  second  printer  multiplied  by  the  factor  for  a 
single  payment  in  three  years.  Algebraically  the  present 
values  determination  of  the  alternatives  is  shown  below. 
(NAVDAC  PUB  15,1980) 

A  =  $10,000  *  ($4,000  X  4.570)  -  $28,280 
B  =  $8,000  ♦  ($5,000  X  4.570)  +  ($8,000  x  . 788)=$37 , 154 
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b.  Present  Value  Using  a  Three  Year  Service  Life 
Had  a  three  year  service  life  been  used  in  the 
comparison  only  one  purchase  in  alternative  B  would  have  to 
have  been  considered.  However,  as  alternative  A  would  have 
half  of  its  service  life  remaining  at  the  end  of  three 
years,  then  half  of  its  purchase  price,  multiplied  by  the 
factor  for  a  single  cash  flow  in  three  years,  would  have  to 
be  deducted  from  the  calculation  to  represent  the  remaining 
value.  Also,  the  reoccurring  expenses  multiplied  by  the 
factor  for  a  three  year  cumulative  uniform  series,  not  a  six 
year  series.  Algebraically  the  present  values  determination 
of  the  alternatives  using  this  three  year  service  life  is 
given  below.  (HAVDAC  PUB  15,1980) 

A  s  $10,000  +  ($4,000  X  2.609)  -  ($5,000  x  .788)s$16,496 
B  =  $8,000  +  ($5,000  X  2.609)  =  $21,045 
3.  Advantages  &  Disadvantages  of  Present  Value  Analysis 
A  key  advantage  of  the  present  value  technique  is 
that  organizations  are  familiar  with  it  for  the  technique  is 
used  to  make  other  capital  budgeting  decisions 
(Garrison, 1988) .  Another  advantage  is  that  present  value 
analysis  is  a  simple  means  of  comparing  alternatives  when 
the  benefits  and  project  lives  are  the  same  (HAVDAC  PUB 
15,1980).  Then  possibly  the  most  compelling  advantage  of 
present  value  analysis  is  that  it  is  one  of  the  three  cost 
analysis  methods  recommended  by  the  GSA  for  use  in  the 
analysis  of  alternatives.  This  analysis  being  required  for 
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each  federal  acquisition  of  ADP  equipment  or  information 
systems . 

Present  value  analysis  alone  is  seen  by  some  as  ill 
suited  for  the  analysis  of  some  types  of  information 
systems,  such  as  decision  support  systems  (Keen, 1981).  The 
technique  is  not  recommended  in  this  situation  for  it  is 
difficult  to  quantify  the  benefits  of  such  a  system 
capabilities  such  as  AD-HOC  analysis  (Keen, 1981).  Present 
value  analysis,  as  described  above,  is  also  criticized  for 
it  does  not  take  into  account  risk  or  the  probability  of  the 
benefit  occurring  (Couger  et  al,1982).  Possibly  for  these, 
or  similar  reasons  GSA  has  also  required  that  non-cost 
factors  be  included  in  the  analysis  of  alternatives. 

D.  ANALYSIS  OF  MON-COST  FACTORS 

The  non-cost  factors  that  the  GSA  has  specified  that  are 
to  be  included  in  an  analysis  of  alternatives  have  been 
separated  by  GSA  into  two  groups;  functional  factors  and 
risk  factors.  The  functional  factors  include:  (U.S. 

GSA, 1990) 

obsolescence 
availability 
reliability 
-  maintainability 
expandability 
flexibility 
security 
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privacy 

personnel  impacts 
user  acceptance 
accountability 

The  risk  factors  are:  (U.S.  GSA,1990) 

financial 

technical 

schedule 

The  inclusion  of  these  functional  and  risk  factors  into 
an  analysis  of  alternatives  offsets  the  disadvantages  of 
conducting  an  analysis  solely  on  the  basis  of  cost  factors. 
Those  disadvantages  namely  being;  the  lack  of  addressing 
risk,  and  the  inappropriate  nature  of  attempting  to  apply 
cost-benefit  analysis  methods  to  benefits  that  are  difficult 
to  quantify  in  dollar  terms.  As  such,  there  should  be  a 
clear  advantage  of  conducting  an  analysis  of  alternatives 
that  incorporates  both  cost  and  non-cost  factors.  However, 
even  though  GSA  does  directs  that  the  functional  and  risk 
factors  be  included  in  the  analysis  of  alternatives,  the  GSA 
guides  do  not  recommend  or  mension  any  non-cost  analysis 
method  which  should  be  used  in  such  an  analysis.  (U.S. 

GSA, 1990) 
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IV.  ANALYSIS  METHODS  FOR  NON-COST  FACTORS 


There  are  existing  benefit  comparison  methods  that  may 
well  suited  for  fulfilling  the  GSA  requirement  for  non-cost 
analysis.  That  analysis,  of  course,  being  part  of  an 
analysis  of  alternatives  leading  up  to  the  federal 
acquisition  of  ADP  equipment  or  information  systems.  These 
benefit  comparison  methods  not  only  are  useful  in 
contrasting  the  benefits  of  the  various  alternatives,  they 
can  also  be  used  for  making  a  selection  from  proposed 
alternatives  based  on  non-cost  factors.  The  non-cost 
factors  in  this  context  may  be  described  as  attributes  of 
information  systems  and  may  be  analyzed  as  well  as  measured 
as  such  by  the  different  benefit  analysis  methods.  The 
supporters  of  objective  quantification  methods  believe  that 
analysts,  who  describe  the  benefits  of  information  systems 
as  completely  intangible,  are  not  putting  forth  the  effort 
to  find  measures  for  the  benefits  (Couger  et  al,1982).  The 
benefit  comparison  methods  that  will  be  addressed  are:  the 
additive  weight  model,  the  eigenvector  model,  and  the  multi¬ 
attribute  utility  model. 

A.  THE  ADDITIVE  WEIGHT  MODEL 
1.  Description 

The  additive  weight  model  consists  of  three  steps; 
determining  weights,  scoring  alternatives,  and  selecting  the 
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best  alternative.  Prior  to  using  the  model  however,  the 
benefits  must  be  defined  and  broken  down  into  the  separate 
attributes  or  characteristics  of  the  information  system. 

Such  attributes  could  include:  memory  capacity,  calculation 
speed,  or  the  manufacturer's  reliability.  These  attributes 
can  be  divided  by  categories,  such  as  hardware  and  software, 
or  listed  together  in  one  group.  The  attributes  are  not 
required  to  be  independent  of  each  other  (Shoval  and 
Lugasi , 1987 ) .  Defining  the  attributes  establishes  common 
references  in  which  the  additive  weight  model  can  be 
implemented  by  analysts  and  decision  makers  alike.  (Shoval 
and  Lugasi, 1987) 

a.  The  Three  Steps  of  the  Additive  ffeight  Model 

The  first  step  determines  attribute  weights  for 
each  attribute.  These  may  be  obtained  through  point 
allocation  where  a  decision  maker  assigns  a  number  between  0 
and  1  to  each  attribute  to  reflect  its  relative  importance 
(Schoemaker  and  Waid,1982).  The  sum  of  all  the  attribute 
importance  weights  must  equal  1.  If  attributes  are  divided 
into  categories  then  category  importance  weights  must  also 
be  selected  and  'heir  sum  must  also  equal  1. 

The  second  step,  scores  alternatives,  by  having 
the  decision  maker  evaluate  each  alternative  and  assess  a 
rating  for  each  attribute.  This  is  done  one  attribute  at  a 
time  and  only  the  attributes  receive  a  rating,  not  the 
categories  or  the  overall  system.  The  final  step,  selection 
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of  the  best  alternative,  is  made  by  choosing  the  information 
system  that  maximizes  the  suimnation  of  attribute  scores 
multiplied  by  the  respective  attribute  and  category  weights. 
This  step  is  expressed  in  Formula  3  and  demonstrated  in 
Table  3.  (Shoval  and  Lugasi,1987) 

b.  The  Additive  freight  Formula 

Formula  3  is  the  additive  weight  formula.  When 
using  this  formula  four  conditions  must  hold  true.  First, 
the  sum  of  the  category  weights  must  equal  1.  Second, 
attribute  weights  range  from  0  to  1.  Third,  attribute 
scores  must  be  less  than  or  equal  to  1.  Finally,  the  sum  of 
the  attribute  importance  weights  must  equal  1.  (Shoval  and 
Lugasi ,1987 ) 

The  following  notation  is  used  in  Formula  3  and 
Tables  1  through  3.  expresses  the  category  importance 
weights  of  m  categories.  fTj  represents  attribute  importance 
weights  of  n  attributes.  Finally  V^j  symbolizes  the  ith 
attribute  score  for  alternative  j.  (Shoval  and  Lugasi, 1987) 

JD  Xtjt 

(3) 


2.  Exaoqtle  of  the  Additive  Weight  Model 

In  Shoval  and  Lugasi  (1987)  an  example  of  the 
additive  weight  model  is  given  where  there  are  four 
alternatives  to  select  from  and  14  attributes  to  be 
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evaluated  which  were  split  into  three  categories.  The  four 
proposed  systems  are  alternatives  A  through  D.  The  three 
categories  are  hardware,  software,  and  support.  The 
attributes,  listed  by  category,  are  given  in  Table  1  along 
with  their  respective  importance  weights.  Alternative 
scores  by  attribute  are  given  in  Table  2.  (Shoval  and 
Lugasi ,1987 } 


Table  1.  Category  and  Attribute  Importance  Weights 


Category  Attribute 

Weight 

Z1 

Hardware 

.30 

XI  Memory  Capacity 

.25 

X2  Calculation  Speed 

.25 

X3  I/O  Speed 

.07 

X4  Equipment  Reliability 

.25 

X5  Flexibility  for  Expansion 

.18 

Z2 

Software 

.30 

X6  Availability  for  Scientific 

Software  .30 

X7  Flexibility  to  Changes 

.30 

X8  Software  Performance 

.40 

Z3 

Support 

.40 

X9  Manufacturer's  Reliability 

.20 

XIO  Supplier's  Reliability 

.20 

XII  Supply  Time 

.10 

X12  Quality  of  Hardware  Support 

.20 

XI3  Quality  of  Software  Support 

.20 

X14  Quality  of  Documentation 

.10 
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Table  2.  Alternative  Scores  by  Attribute 


Table  3.  Calculation  of  the  Weighted  Score  for 
Alternative  A 


Attribute 

Category  x 
Weight 

Attribute 

Weight 

X  Attribute 
Score 

-  Weighted 
Score 

XI 

0.3 

.25 

0.8 

.0600 

X2 

0.3 

.25 

0.8 

.0600 

X3 

0.3 

.07 

0.8 

.0168 

X4 

0.3 

.25 

1.0 

.0750 

X5 

0.3 

.18 

1.0 

.0540 

.2658 

X€ 

0.3 

0.3 

0.8 

.0720 

X7 

0.3 

0.3 

1.0 

.0900 

X8 

0.3 

0.4 

0.9 

.1080 

.2700 

X9 

0.4 

0.2 

1.0 

.0800 

XIO 

0.4 

0.2 

0.9 

.0720 

XU 

0.4 

0.1 

0.9 

.0360 

X12 

0.4 

0.2 

1.0 

.0800 

XI 3 

0.4 

0.2 

0.9 

.0720 

X14  0.4 

Total  Weighted  Score 

0.1 

1.0 

.0400 

.3800 

.9158 
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From  Formula  3  and  Tables  1  through  3,  alternative 

A,  with  a  score  of  0.9158,  has  the  highest  total  weighted 
score  of  the  four  alternatives.  Alternative  A  would  thus  be 
the  information  system  selected  by  the  additive  weight 
model.  The  weighted  scores  of  alternatives  B,  C,  and  0  are 
0.9024,  0.8399,  and  0.8557  respectively.  (Shoval  and 
Lugasi,1987) 

3.  Advantages  and  Disadvantages  of  Additive  Height 

Besides  being  simple  in  nature  the  additive  weight 
model  also  has  the  advantage  that  sensitivity  analysis,  or 
receptiveness  to  change,  can  be  measured  for  the  weights 
assigned  to  the  various  attributes  and  categories.  The 
disadvantage  is  that  this  model  cannot  check  the  consistency 
of  evaluators;  nor  does  it  consider  risk  or  uncertainty. 
Assigning  weights  and  scoring  alternatives  is  also  a 
subjective  appraisal  of  information  system  benefits. 

(Shoval  and  Lugasi,1987) 

B.  THE  EIOEHVECTOR  MODEL 
1 .  Description 

Building  on  the  additive  weight  model  the 
eigenvector  model  provides  a  means  for  determining  the 
attribute  and  category  weights  as  well  as  scoring  the 
alternatives.  Final  system  selection  in  the  eigenvector 
model  is  made  using  Formula  3  of  the  additive  weight  model . 
In  the  eigenvector  model  attribute  weights  are  found  by  a 
process  which  starts  with  pairwise  comparisons.  A  decision 
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maker  completes  a  matrix  in  which  both  the  number  of  columns 
and  rows  equal  the  number  of  attributes  being  evaluated. 

For  instance,  if  four  attributes  were  being  examined,  then  a 
4x4  matrix  would  be  required.  Each  cell  of  the  matrix 
represents  a  comparison  between  two  of  the  attributes.  A 
nine-point  scale  is  used  in  the  model  with  1  representing 
equality  between  attributes  and  9  depicting  absolute 
preference  of  one  attribute  over  the  other.  In  the  matrix 
in  Figure  1,  attribute  xl  is  favored  3/2  times  to  attribute 
x5.  The  consistency  of  the  decision  maker  can  be  verified 
through  an  inspection  of  inverse  relationships.  Here  x5  is 
favored  2/3  times  to  xl  so  the  decision  maker  is  being 
consistent.  (Shoval  and  Lugasi ,1987) 

Figure  1.  Sample  Matrix 


Xl 

X2 

X3 

X4 

X5 

Xl 

1 

1 

5/2 

1 

3/2 

X2 

1 

1 

5/2 

1 

3/2 

X3 

2/5 

2/5 

1 

1/3 

1/2 

X4 

1 

1 

3 

1 

3/2 

X5 

2/3 

2/3 

2 

2/3 

1 

The  eigenvector  of  this  matrix  is  then  computed  for 
the  maximum  eigenvalue.  This  eigenvector  is  then  normalised 
so  that  the  sum  of  the  elements  in  the  vector  equals  1. 
Computing  and  normalising  these  eigenvectors  can  be  done 
through  the  use  of  a  computer  package  such  as  El SPACE,  which 
is  a  package  of  fortran  IV  programs  (Coos  and 
Hartmanis,1977) .  The  elements  in  this  final  form  of  the 
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eigenvector  are  the  respective  attribute  weights.  Category 
weights  are  then  determined  in  a  similar  fashion  with  the 
dimensions  of  the  matrix  equal  to  the  number  of  categories. 
(Shoval  and  Lugasi,1987) 

To  score  the  alternatives  by  attribute,  a  matrix  of 
pairwise  comparisons  is  used.  This  time  the  number  of 
matrices  used  is  equivalent  to  the  number  of  attributes. 

The  dimensions  of  each  of  these  matrices  is  set  to  the 
number  of  alternatives.  The  elements  of  the  normalised 
eigenvector  are  the  solution.  Here  they  represent  the 
scores  of  the  alternatives  for  the  respective  attributes. 
(Shoval  and  Lugasi,1987) 

2.  Example  of  the  Eigenvector  Model 

The  situation  used  in  this  example  is  the  same  as 
that  used  above  for  the  additive  weight  model :  four 
alternatives,  three  categories,  and  14  attributes.  The  four 
alternatives  are  still  labeled  A  through  D  and  the 
categories  and  attributes  remain  the  same  from  Table  1 
above.  In  Figure  2  of  Appendix  A  the  matrices  for 
determining  attribute  weights  are  displayed.  Figure  3  of 
Appendix  A  is  the  matrix  of  pairwise  comparisons  of  the 
three  categories.  In  Figure  4  of  Appendix  A  the  14  matrices 
are  presented  that  were  used  to  define  the  alternative 
scores  by  attribute.  Finally,  Table  4  contains  the  category 
and  attribute  weights,  and  lists  the  weighted  scores  for  the 
each  alternatives  which  is  used  to  produce  their  respective 
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ranking  as  determined  by  the  eigenvector  model.  (Shoval  and 
Lugasi,1987) 


Table  4.  Scores  and  Ranking  of  Alternatives 


Attribute 

Weight 

A 

Alternatives 
B  C 

D 

Z1 

.333 

XI 

.244 

.222 

.222 

.222 

.333 

X2 

.244 

.286 

.286 

.286 

.143 

X3 

.091 

.211 

.211 

.199 

.378 

X4 

.253 

.316 

.316 

.158 

.210 

X5 

.169 

.286 

.286 

.286 

.143 

Z2 

.333 

X6 

.400 

.286 

.286 

.286 

.143 

X7 

.200 

.250 

.250 

.250 

.250 

X8 

.400 

.273 

.273 

.273 

.182 

Z3 

.333 

X9 

.352 

.375 

.250 

.125 

.250 

XIO 

.235 

.316 

.284 

.142 

.258 

XU 

.062 

.275 

.347 

.277 

.102 

X12 

.139 

.351 

.351 

.109 

.189 

X13 

.139 

.333 

.333 

.111 

.222 

X14 

.072 

.315 

.315 

.153 

.216 

Weighted 

Score 

.295 

.280 

.213 

.212 

Ranking 

1 

2 

3 

4 

3.  Advantages  &  Disadvantages  of  the  Eigenvector  Model 
The  advantage  of  the  eigenvector  model  is  that  it 
facilitates  a  review  of  a  decision  makers  consistency 
throughout  the  analysis  by  conducting  an  inspection  of 
inverse  relationships  between  respective  paired  comparisons. 
However,  the  eigenvector  model  ignores  attribute 
interdependence  and  does  not  reflect  risk  or  uncertainty 
involved  with  the  various  alternatives  in  the  analysis.  In 
the  course  of  a  large  scale  analysis  pairwise  comparisons 
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can  also  become  quite  difficult  when  the  number  of  pairs  is 
increased  to  a  larqe  ntimber.  (Shoval  and  Lugasi,1987) 

C.  THE  HOLTI-ATTRIBUTE  UTILITY  MODEL 
1.  Description 

Like  the  additive  weight  and  eigenvector  models  the 
multi-attribute  utility  model  requires  a  list  of  applicable 
attributes  to  be  evaluated.  While  benefits  of  information 
systems  are  not  unidimensional,  it  is  probable  that 
dimensionality  can  be  contained  or  at  least  reduced  by 
including  only  the  most  relevant  attributes  to  the  systems 
being  evaluated  (Ahituv,1980) .  The  multi-attribute  utility 
model  itself  has  two  variations,  an  additive  model  and  a 
multiplicative  model  (Formulas  4  and  5  respectively).  The 
variants  differ  in  the  way  they  treat  risk.  (Shoval  and 
Lugasi ,1987) 

a.  Multi-Attribute  Utility  Fonsulas 

(4) 


a 


k 


(5) 


In  both  Formulas  4  and  5,  Uj  depicts  the  utility 
of  alternative  j.  U^j  refers  to  the  utility  of  attribute  i 
in  alternative  j  and  U  itself  is  constrained  to  be  between  0 
and  1  by  the  constant  k.  Formula  6  below  is  used  to 
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determine  k.  here  portrays  the  weiyht  of  attribute  i. 
(Shoval  and  Lu9asi,1987) 


(6) 


b.  Model  Selection 

The  use  of  either  the  additive  or  multiplicative 
model  is  dependent  on  the  decision  maker's  outlook  on  risk. 
The  decision  aid  for  that  determination  is  the  summation  of 
all  the  attribute  weights.  If  this  sum  is  equal  to  1,  then 
the  decision  maker  is  indifferent  to  risk  and  the  additive 
model  is  selected.  If  the  sum  is  greater  than  1/  then  the 
multiplicative  model  is  selected  and  k  set  between  0  and  -1. 
When  the  summation  of  attribute  weights  is  less  than  1,  then 
the  multiplicative  model  is  also  used,  yet  in  this  case  k  is 
set  to  a  value  greater  than  0.  (Shoval  and  Lugasi,1987) 

c.  JIpplication  of  the  Multi-Attribute  Utility  Model 
The  multi -attribute  utility  model  requires  both 

utility  and  preference  independence  of  the  various 
attributes.  Once  the  attribute  independence  has  been 
confirmed  the  evaluation  of  the  utility  function  for  every 
attribute  can  then  be  based  on  two  axioms,  transitivity  and 
continuity.  The  transitivity  axiom  states  that  if 
alternative  a^  is  favored  over  Mj,  and  a^  is  favored  over  a^, 
then  alternative  a^  is  superior  to  alternative  aj|.  The 
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continuity  axiom  refers  to  the  state  that  if  alternative  a^ 
is  favored  over  a^,  and  mj  is  favored  over  a^,  then  a  gamble 
on  a^  and  a^  can  be  performed  so  that  there  will  be  a  point 
where  the  decision  maker  will  be  indifferent  between 
choosing  the  gamble  or  being  assured  of  receiving 
alternative  a^.  If  the  axioms  can  be  met  then  the  utility 
function  can  be  found  through  the  multi-attribute  utility 
model.  (Shoval  and  Lugasi,1987) 

d.  Forming  the  Utility  Function 

To  form  the  utility  function  for  each  attribute, 
with  the  above  axioms  previously  being  met,  the  two  extreme 
points  of  the  utility  curve  must  be  defined.  The  highest 
level  for  attribute  i  is  designated  as  i*  and  U(i*)  is  set 
to  equal  1.  The  lowest  level  of  attribute  i  is  represented 
by  i^  and  U(i^)  is  set  to  0.  i  is  designated  as  i*  when 
conducting  the  gambling  technique  in  evaluating  the  utility 
of  a  particular  level  of  i.  The  decision  maker  is  then 
presented  with  the  following  situation.  Given  two 
alternatives,  A  and  B,  with  all  attributes  between  the  two 
alternatives,  other  than  i,  being  equal.  In  alternative  A 
there  is  a  100%  probability  that  attribute  i  will  be  i*. 
Alternative  B  is  the  gamble  with  a  probability  of  p  that 
attribute  i  will  be  i*  and  a  probability  of  1-p  that  i  will 
be  i*.  The  decision  maker  is  then  asked  to  determine  at 
what  level  of  p  he  would  be  indifferent  between  the  two 
alternatives.  When  the  decision  maker  is  indifferent  to  the 


two  alternatives  the  utility  of  alternative  A  equals  the 
expected  utility  of  alternative  B,  or  in  other  words  U(i')  - 
pO(i*)  +  (l-p)U(i*).  The  value  of  V(i')  is  then  known  to  be 
p,  for  0(i*)  is  1  and  U(i^)  is  0.  Other  utility  points  for 
attribute  i  are  found  in  the  same  manner  by  varying  the 
level  of  i ' .  The  utility  curve  for  i  is  formed  by 
connecting  the  utility  points.  A  continuous  curve  is  not 
required  for  it  is  suitable  to  evaluate  only  a  range  of 
relevant  values  for  each  attribute.  (Shoval  and 
Lugasi , 1987 ) 

e.  Determining  the  Attribute  Meigbta 

The  attribute  weights,  kj,  are  found  in  the  same 
way  that  the  attribute  utility  points  were  found.  To 
define  the  attribute  weights  the  decision  maker  is  given  a 
situation  with  two  alternatives,  A  and  B.  This  time 
alternative  A  has  attribute  i  at  the  highest  level,  or  i*, 
and  the  remaining  attributes  at  their  lowest  level,  or  iK 
Alternative  B  is  again  a  gamble  with  a  probability  of  p  that 
all  of  the  attributes  are  at  their  respective  highest  level, 
and  a  probability  of  1-p  that  all  of  the  attributes  are  at 
their  lowest  level.  The  decision  maker  is  then  asked  to 
choose  between  the  two  alternatives  by  answering  the 
following  question:  ”At  what  level  of  p  would  you  be 
indifferent  between  the  two  alternatives?”  The  value  of  k^ 
is  then  known  to  be  p  by  the  same  reasoning  that  D(i')  is 
equal  to  p.  Hith  the  utility  function  and  weights  found  for 
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every  attribute  the  appropriate  version  of  the  model  can 
then  be  applied.  (Shoval  and  Lugasi,1987) 

2.  Example  of  the  Multi -At tribute  Utility  Model 

The  example  cited  here  is  from  Shoval  and  Lugasi 
(Shoval  and  Lugasi, 1987)  and  is  the  same  situation  as  given 
in  the  examples  for  the  additive  weight  and  eigenvector 
models . 

a.  Determining  Attribute  Independence 

Prior  to  the  application  of  the  multi-attribute 
utility  model  the  attribute  utility  and  preference 
independence  had  to  be  verified.  The  utility  independence 
was  confirmed  through  the  repeated  use  of  the  gambling 
technique.  In  each  trial  the  level  of  the  attributes  other 
than  i  were  set  to  different  levels.  Jtility  independence 
was  known  to  be  true  as  D(i')  remained  constant  throughout 
the  trials.  (Shoval  and  Lugasi, 1987) 

In  a  similar  manner  the  preference  independence 
was  also  found.  Attributes  were  evaluated  taking  them  two 
at  a  time.  In  the  first  alternative,  the  first  attribute, 
quality  of  documentation,  was  set  at  2  out  of  a  five-point 
scale  and  the  second  attribute,  memory  capacity,  was  placed 
at  768K.  The  second  alternative  had  quality  of 
dociimentation  at  4  and  memory  capacity  at  i ' .  All  other 
attributes  were  at  their  highest  respective  levels.  The 
decision  maker  was  then  asked  to  determine  a  value  for  i ' 
where  he  would  be  indifferent  between  the  two  alternatives. 
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This  process  was  repeated  with  all  of  the  attributes  other 
than  quality  of  documentation  and  memory  capacity  set  to 
their  lowest  respective  levels.  As  1'  was  found  not  to 
change  between  the  two  trials,  the  preference  between  the 
two  attributes  was  determined  to  be  independent.  The  rest 
of  the  attribute  pairs  were  examined  in  the  same  way  and 
found  to  be  preference  independent.  (Shoval  and 
Lugasi , 1987 ) 

b.  Determining  Attribute  Otility 

In  evaluating  the  utility  of  the  attributes  the 
categories  were  ignored  and  the  14  attributes  were  addressed 
in  one  list.  Continuous  utility  curves  were  not  calculated 
for  it  was  determined  that  for  each  attribute  only  four 
values,  one  for  each  alternative,  needed  to  be  evaluated. 

To  evaluate  the  memory  capacity  the  decision  maker  was  again 
given  two  alternatives.  Alternative  A  is  certain  to  receive 
768K  capacity.  Alternative  B  is  a  gamble  with  the 
probability  of  p  that  the  capacity  will  be  1060K  and  a 
probability  of  1-p  that  the  capacity  will  be  576K.  The 
minimvun  acceptable  memory  capacity  having  previously  been 
defined  at  576K  and  the  highest  capacity  between  all 
alternatives  being  1060K.  After  asking  the  decision  maker 
for  a  value  of  p  where  he  would  be  indifferent  between  the 
two  alternatives  he  responded  that  p  would  have  to  be  0.3 
for  him  to  be  indifferent.  Thus  the  memory  capacity  of  768K 
has  the  utility  value  of  0.3.  The  utility  values  of  all  of 


33 


the  14  attributes  were  found  using  this  same  method.  The 
utilities  found  for  each  of  the  attributes  of  each  of  the 
four  alternatives  are  presented  in  Table  5.  (Shoval  and 
Lugosi, 1987) 

c.  Determining  Attribute  Heights 

Attribute  weights,  k^,  were  found  as  in  the 

following  example  for  memory  capacity.  The  decision  maker 
being  provided  with  two  alternative  computer  systems.  One 
has  a  guaranteed  capacity  of  1060K  and  all  the  remaining 
attributes  are  at  their  lowest  levels.  The  other 
alternative  is  the  gamble  with  a  probability  of  p  that  all 
attributes  are  at  their  highest  levels  and  a  probability  of 
1-p  that  all  of  the  attributes  are  at  their  lowest  levels. 
The  decision  maker  was  then  asked  for  a  value  of  p  where  he 
would  feel  indifferent  between  the  two  alternatives.  The 
response  was  that  p  would  have  to  be  0.08  for  the  decision 
maker  to  be  indifferent.  Thus  the  weight  of  the  attribute 
memory  capacity  is  0.08.  The  weights  for  the  other 
attributes  were  found  in  the  same  manner  and  are  given  in 
Table  5.  (Shoval  and  Lugasi,1987) 

d.  Implementing  the  Model 

As  the  sum  of  the  weights  was  greater  than  1  the 
multiplicative  utility  model.  Formula  5,  was  used  in  this 
example.  The  constant  k  was  found  to  be  -0.44268  from 
Formula  6.  The  ranking  of  the  alternatives  is  given  in  the 
last  line  of  Table  4.  (Shoval  and  Lugasi,1987) 
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Table  5.  Utility  Values  and  Heights 


Attribute 

Height 

A 

Alternatives 

B  C 

D 

XI 

0.08 

0.6 

0.6 

0.6 

0.1 

X2 

0.07 

1.0 

1.0 

1.0 

0.5 

X3 

0.04 

0.9 

0.9 

0.8 

1.0 

X4 

0.12 

1.0 

0.9 

0.7 

0.9 

X5 

0.07 

1.0 

0.9 

1.0 

0.8 

X6 

0.07 

1.0 

1.0 

1.0 

0.8 

X7 

0.15 

1.0 

1.0 

1.0 

1.0 

XS 

0.10 

1.0 

1.0 

1.0 

0.9 

X9 

0.15 

1.0 

0.9 

0.7 

0.8 

XIO 

0.10 

1.0 

0.9 

0.6 

0.9 

Xll 

0.05 

0.8 

0.7 

1.0 

0.9 

XI 2 

0.09 

1.0 

1.0 

0.9 

1.0 

X13 

0.08 

1.0 

1.0 

0.7 

0.8 

XI 4 

0.12 

1.0 

1.0 

1.0 

0.9 

Utility 

.973 

.944 

.882 

.905 

Ranking 

1 

2 

4 

3 

3.  Advantages  &  Disadvantages  of  Multi-Attribute 

Utility 

There  are  three  primary  advantages  of  the  multi - 
attribute  utility  model.  First,  the  model  is  normative  in 
that  it  is  based  on  the  axioms  of  transitivity  and 
continuity  which  reveal  the  behavioral  rules  of  the  decision 
maker.  Secondly,  the  model  verifies  attribute  independence. 
Finally,  the  model  differs  from  the  additive  weight  and 
eigenvector  models  in  that  it  has  the  capacity  to  take  into 
account  the  issues  of  risk  and  uncertainty.  (Shoval  and 
Lugasi , 1987 ) 

The  disadvantages  of  the  multi-attribute  utility 
model  are  two  fold.  First,  the  decision  maker  may  have 
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difficulty  with  the  gambling  technique  in  making  a 
preference  between  the  two  alternatives.  Secondly,  the 
rules  of  attribute  independence,  necessary  for  the 
implementation  of  the  model,  are  not  always  met.  Thus  there 
are  situations  were  the  model  can  not  be  used.  (Shoval  and 
Lugasi,1987) 

D.  SUMMARY 

In  sumnary  the  benefit  comparison  methods;  additive 
weight,  eigenvector,  and  multi-attribute  utility,  are  all 
similar  in  nature.  All  three  methods  start  by  defining  the 
various  advantageous  features,  or  non-cost  factors,  of  an 
information  system  that  are  to  be  evaluated  and  separate 
those  features  into  distinct  attributes.  Then  each 
attribute  is  assigned  either  an  importance  weight  or 
relative  utility,  dependent  on  the  analysis  method.  A 
score,  or  utility,  is  next  found  for  each  alternative  by 
attribute.  Prom  these  scores  the  various  formulas  of  the 
different  methods  can  then  be  used  to  arrive  at  a  final 
ranking  of  each  alternative.  The  methods  differ,  of  course, 
by  the  formulas  used  and  if  any  special  conditions  have  to 
be  met,  such  as  attribute  independence  in  the  multi¬ 
attribute  utility  model. 

1.  Advantages  &  Disadvantages  of  Benefit  Coaiparison 
All  of  the  methods  addressed  in  this  chapter  have 
their  own  merits.  The  additive  weight  model  has  the 
capacity  for  sensitivity  analysis  however,  in  the 
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application  of  the  model  in  the  example,  the  model  does  not 
take  into  account  risk  or  imcertainty.  The  eigenvector 
model  has  the  capacity  to  verify  a  decision  maker's 
consistency,  yet  it  too  does  not  consider  risk  or 
uncertainty.  In  addition,  the  eigenvector  model  overlooks 
the  issue  of  attribute  interdependence.  The  multi-attribute 
utility  model  has  the  advantages  that  it  is;  normative,  it 
examines  attribute  independence,  and  it  takes  into  account 
risk  and  uncertainty.  The  disadvantage  of  the  multi - 
attribute  utility  model  is  that  the  gambling  technique,  used 
in  the  differentiation  of  alternatives,  may  be  difficult  for 
some  decision  makers.  The  multi-attribute  utility  model 
also  can  not  be  applied  in  all  situations  particularly  when 
the  rules  of  attribute  independence  can  not  be  met. 

2.  Benefit  Comparison  and  Hon-Cost  Factor  Analysis 
The  different  functional  and  risk  factors 
recommended  by  GSA  could  be  selected  as  the  attributes  to  be 
evaluated  in  the  application  of  a  benefit  comparison  method 
for  the  analysis  of  the  non-cost  factors.  However,  by 
including  the  functional  factor  of  user  acceptance  in  any 
one  of  the  models  would  require  a  scoring  technique  to 
assess  user  satisfaction  with  the  proposed  alternative. 

None  of  the  benefit  comparison  methods  discussed  here 
address  how  such  a  score  could  be  determined.  Thus  there 
remains  a  need  for  an  analysis  method  or  tool  that  can  be 
applied  in  the  evaluation  of  user  acceptance  with  respect  to 


37 


the  analysis  of  alternatives.  Tools  that  have  been 
recommended  for  quantifying  user  acceptance  fall  under  the 
heading  of  measurements  of  user  satisfaction,  which  is  the 
subject  of  Chapter  V. 
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V.  MEASURIRO  USER  SATISFACTIOH 


In  the  preceding  chapter  the  measurement  of  user 
satisfaction  was  introduced  as  a  means  of  augmentation  to 
the  benefit  comparison  methods  so  that  all  of  the  OSA 
stipulated  non-cost  factors  may  be  evaluated  in  an  analysis 
of  alternatives  in  the  acquisition  of  ADP  systems  for  the 
federal  government.  The  supporters  of  user  satisfaction 
analysis  techniques  believe  so  strongly  in  their  work  that 
they  consider  that  these  techniques  should  be  used  more  than 
just  for  augmentation  of  other  evaluation  methods 
(Baroudi , Ives, and  01 son, 1983).  These  defenders  of  user 
satisfaction  regard  the  efforts  to  measure  actual 
productivity  benefits  of  information  systems  as  difficult 
and  even  futile  (Baroudi , Ives, and  Olson, 1983).  Others 
insist  that  when  dealing  with  information  systems  it  is 
impossible  to  derive  useful  measures  such  as  return  on 
investment  or  pretax  profits  (Olson, and  Heill,1989).  User 
satisfaction,  in  the  view  of  these  authors,  is  measurable 
and  for  that  fact  should  be  used  as  the  method  of  choice  in 
the  analysis  of  information  systems  benefit  (Baroudi,  Ives, 
and  Olson, 1983).  Methods  that  can  be  used  in  the  analysis 
of  user  satisfaction  that  will  be  addressed  are:  the  Bailey 
and  Pearson  tool,  the  Electre  model,  and  the  multi-attribute 
linear  value  model. 
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A.  USER  8ATI8FACTI0E  AHALT8IS  METH0D8 


Primarily  as  a  substitute  to  the  objective  methods  of 
information  system  benefit  analysis,  such  as  net  present 
value,  a  subjective  measure  of  user  satisfaction  has  been 
offered.  The  technique  of  user  satisfaction,  also  known  as 
user  information  satisfaction,  centers  around  the  use  of 
surveys  or  questionnaires  to  measure  the  satisfaction  level 
that  the  users  have  with  their  information  system. 

Different  types  of  surveys  have  been  suggested,  such  as 
single  item  and  multiple  item.  Surveys  of  single  items  or 
functions  are  of  limited  utility  (Baroudi , Ives,  and 
Olson, 1983).  Multiple  item  surveys,  more  broad  in  scope, 
take  a  better  sample  of  the  users  opinions  of  the 
information  system  and  therefore  dominate  as  preferred 
measurement  techniques  of  user  satisfaction  (Baroudi,  Ives, 
and  Olson, 1983).  Completed  surveys  are  used  to  make 
inferences  about  the  beneficial  value  of  an  information 
system  as  a  whole.  These  deductions  have  been  made  in  the 
past  from  numerical  scales  and  formulas  that  have  been 
devised  by  different  developers  of  user  satisfaction 
measurement  tools. 

B.  THE  BAILET  AHD  PEAR80H  TOOL 

Bailey  and  Pearson  (Bailey  and  Pearson, 1983)  present  a 
multiple  item  survey  and  a  respective  scoring  method  to 
measure  computer  user  satisfaction.  They  identified  39 
different  factors  of  user  satisfaction  (see  Appendix  B) . 
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Each  factor  identified  was  assigned  four  distinct  adjective 
pairs  to  assess  the  factor.  The  documentation  factor,  for 
instance,  was  assigned  the  distinct  adjective  pairs  of: 

-  clear  vs.  hazy 

-  available  vs.  unavailable 

-  complete  vs.  incomplete 

-  current  vs.  obsolete 

In  addition  to  the  four  distinct  pairs  of  each  factor  the 
adjective  pair  satisfactory  -  unsatisfactory  was  included  as 
a  consistency  check  on  the  other  adjective  pairs.  Finally 
the  adjective  pair  important  -  unimportant  was  included  to 
assign  weights  of  importance  to  the  various  factors  (see 
Appendix  B).  (Bailey  and  Pearson,  1983) 

1.  Numerical  Scales 

To  quantify  the  results  of  the  survey  Bailey  and 
Pearson  (Bailey  and  Pearson, 1983)  derived  their  own 
importance  and  satisfaction  scales.  The  importance  scale, 
used  to  designate  weights  of  importance  to  the  respective 
factors,  was  assigned  a  value  range  of  0.10  to  1.00  in  0.15 
increments.  The  values  were  associated  with  importance  and 
unimportance  as  shown  below: 

1.0  extremely  important 
.85  quite 
.70  slightly 
.55  equally 
.40  slightly 
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.25  quite 

.10  extremely  unimportant 
The  satisfaction  scale  below  was  used  to  assign 
values  to  the  first  four  adjective  pairs  of  each  factor 
(Bailey,  and  Pearson,  1983): 

3  extremely  satisfied 
2  quite 
1  slightly 
0  equally 
-1  slightly 
-2  quite 

-3  extremely  dissatisfied 
2.  Bailey  and  Pearson's  Formula 

Through  the  use  of  the  importance  and  satisfaction 
scales  the  perception  of  a  user  towards  the  respective 
information  system  can  be  captured  as  a  numerical  score. 
This  value  is  obtained  by  first  taking  a  simple  arithmetic 
mean  of  the  values  I  assigned  to  each  of  the  first  4 
adjective  pairs  k  of  each  of  the  factors  j  using  the 
satisfaction  scale.  Then  the  summation  of  the  individual 
factorial  scores  multiplied  by  their  respective  weights  K, 
from  the  importance  scale,  produces  the  overall  measure  of 
satisfaction  S  for  each  user  i  as  seen  in  Formula  7. 
(Bailey,  and  Pearson,  1983) 
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(7) 


Formula  7  is  then  normalised  to  reduce  the 
undesirable  effects  of  any  factors  assessed  as  neutral  or 
meaningless.  Any  factor  is  eliminated  from  consideration  if 
all  four  of  the  respective  distinct  adjective  pairs  were 
evaluated  as  equal  or  sero.  (Bailey,  and  Pearson,  1983) 

3.  Disadvantages  of  Bailey  and  Pearson's  tool 

The  tool  developed  by  Bailey  and  Pearson  takes  a 
broad  look  at  the  information  system  being  evaluated  and  the 
organizational  setting  in  which  it  was  used.  However,  some 
problems  have  been  identified  with  the  manner  in  which  the 
tool  was  developed.  First,  a  sample  sise  of  only  29  was 
used  to  validate  the  tool.  Second,  each  of  the  29  subjects 
had  participated  in  the  development  of  the  tool  (Baroudi, 
Ives  and  Olson, 1983).  Baroudi,  Ives  and  Olson  (Baroudi, 

Ives  and  Olson, 1983)  also  found  that  Bailey  and  Pearson's 
weighted  and  unweighted  results  were  highly  correlated  and 
as  such  the  adjective  pair  important  -  unimportant  was 
txnnecessarily  included  in  the  questionnaire.  This  means 
that  questions  concerning  the  importance  of  individual 
factors  did  not  have  a  significant  impact  on  the  final 
results  of  the  measurement  tool.  A  key  problem  with  the 
tool  is  that  it  violates  measurement  theory  by  treating 
ordinal  measures,  the  adjective  pairs,  as  interval  measures 
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such  as  distance  (Oalletta  and  Lederer ,1989) .  In  the 
application  of  Formula  7  Bailey  and  Pearson  are  conducting 
parametric  statistical  analysis  on  ordinal  measures  which  is 
a  breach  measurement  theory  (Oalletta  and  Lederer ,1989) . 

C.  THE  MDLTI'-ATTRIBUTE  LINEAR  VALUE  PROCEDURE 

A  procedure  that  has  been  employed  in  an  attempt  to 
assess  the  value  of  information  systems  is  the  multi¬ 
attribute  linear  value  (MALV)  technique  (Epstein  and 
King, 1983).  This  application  of  MALV  produces  more  than  a 
ranking  of  alternatives,  similar  to  the  Bailey  and  Pearson 
tool ,  and  though  very  objective  sounding  in  name  this  use  of 
MALV  is  in  actuality  a  subjective  measure  of  user 
satisfaction  for  the  data  manipulated  by  the  procedure  was 
gained  through  the  user  survey  process  (Epstein  and 
King, 1983) . 

1.  Description 

The  application  of  the  MALV  procedure  by  Epstein  and 
King  uses  a  survey,  numerical  scales,  and  a  formula  in  a 
similar  manner  as  Bailey  and  Pearson’s  measurement  tool  of 
user  satisfaction.  MALV  in  this  application  receives  data 
through  the  use  of  a  ten  item  survey.  Each  item  is  a 
factor,  or  attribute,  of  information  value.  All  ten 
attributes  were  selected  by  the  model  designer.  The  ten 
attributes  are:  (Epstein  and  King, 1983) 

Reporting  cycle 
Sufficiency 
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Understandability 
Freedom  from  bias 
Reporting  delay 
Reliability 
Decision  relevance 
Cost  efficiency 
-  Comparability 
Quantitativeness 

2.  Epstein  and  King's  numerical  Scales 

Epstein  and  King  then  used  a  scale  to  attach 
numerical  scores  to  each  of  the  attributes.  The  scale 
applied  to  the  reporting  cycle  measures  the  frequency  that 
information  inputs  have  to  be  generated  for  user's  decision 
making  (Epstein  and  King, 1983): 

Reporting  Cycle  Attribute  Scale 


Greatest  Satisfaction  Possible  100 
Extremely  frequent  90 
Fairly  infrequent  70 
Moderately  frequent  50 
Very  frequent  30 
Very  infrequent  10 
Lowest  Satisfaction  Possible  0 


Relative  importance  weights  are  obtained  by  a  graphic 
scale  on  the  survey  where  the  response  positions  are 
stressed  and  not  the  scale  values.  Survey  evaluators  can 
then  quantify  the  weights  by  assigning  values  that  match  the 
distance  ratios  between  points  and  the  origin.  (Epstein  and 
King, 1983) 
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3.  MALV  Fornula 


To  find  the  overall  value  V  of  an  information  system 
the  multi-attribute  linear  value  model  simply  multiplies 
each  attribute  score  V(a)  by  the  respective  importance 
weight  K  and  adds  together  all  of  the  results.  Formula  8 
summarizes  this  process.  (Epstein, and  King, 1983) 

10 

(8) 


4.  Siibjectivity  in  Disguise 

This  application  of  MALV  is  essentially  a  subjective 
measure  of  user  satisfaction  as  the  data  are  acquired 
through  the  use  of  a  ten  item  survey.  The  attribute  scales 
and  the  formula  of  MALV  conflict  with  measurement  theory  in 
the  same  manner  as  Bailey  and  Pearson's  tool.  The  model 
fits  adjectives  to  a  numerical  scale  and  then  analyzes  the 
results  as  if  they  were  interval  level  measures.  The 
validity  of  the  model  results  are  questionable  in  that  they 
are  derived  from  the  manipulation  of  ordinal  measures  and 
the  only  legitimate  statistical  analysis  on  ordinal  measures 
is  nonparametric,  such  as  rank  ordering  (Oalletta  and 
Lederer,1989) . 
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D.  THE  ELECTRE  MODEL 


1.  Description 

A  second  analysis  method  of  user  satisfaction  that 
can  be  used  to  evaluate  completed  user  satisfaction  surveys 
is  the  Electre  model  of  multiple  criteria  decision  making. 
This  model  is  based  on  pairwise  comparisons  of  alternatives, 
i  and  j,  and  employs  two  indexes  to  determine  a  final 
ranking  of  various  alternatives  (Zeleny ,1982) .  The  first 
index  is  the  concordance  index  C(i,j)  which  is  the  summation 
of  the  corresponding  criteria  weights  for  all  criteria  in 
which  i  out  performs  j,  divided  by  the  summation  of  all  the 
criteria  weights  of  the  model  (Bui, 1981).  The  second  index 
is  the  discordance  index  D(i,j)  which  is  found  by  taking  the 
largest  difference  in  evaluation  scores  for  which  j  outranks 
i  and  dividing  that  value  by  the  maximum  range  between 
highest  and  lowest  possible  scores  (Bui, 1981).  The  two 
indexes  are  then  used  in  conjunction  with  threshold  values 
selected  arbitrarily  by  the  user.  The  concordance  threshold 
P  ranges  from  0.5  to  1.0  and  increases  in  severity  as  it 
approaches  1.  The  discordance  threshold  Q  ranges  from  0  to 
1  and  increases  in  severity  as  it  approaches  0.  The  final 
ranking  of  alternatives  is  made  as  follows  (Bui, 1981): 


11 

C(i,j)  >=  P  and  D(i,j)  <=  Q 
C(j,i)  >=  P  and  D(j,i)  <~  Q 
i  outranks  j  and  j  outranks  i 
In  all  other  cases 


Tbea 

i  outranks  j 
j  outranks  i 
i  and  j  are  equal 
i  and  j  are  incomparable 
or  the  comparison  is  not 
determinable. 
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2.  Example 

a.  Applying  the  Eiiectre  Model 

If  the  Electre  model  were  applied  to  the 
analysis  of  user  satisfaction  for  three  alternatives,  the 
first  issue  to  be  settled  would  be  to  select  the  different 
criteria  of  user  satisfaction  that  would  be  evaluated  in  the 
analysis.  Along  with  the  criteria,  a  weight  of  relative 
importance  for  each  criterion  must  also  be  determined.  For 
this  example  the  following  criteria  and  weights  will  be 
used: 

Understandability  of  system  requests  and  responses  (.2) 
Ease  of  learning  the  new  system  (.3) 

-  Key  mapping,  or  function  keys  having  the  same  use  as  in 
other  applications  used  by  the  same  user  (.1) 
Availability  of  the  system  to  the  user  (.1) 

System  response  time  (.3) 

A  second  requirement  in  setting  up  the  Electre 
model  is  selecting  a  scale  for  use  in  the  application  of  the 
model.  For  this  example  the  following  scale  will  be  used: 


Greatest  Satisfaction  Possible  100 
Extremely  Satisfied  90 
Fairly  Satisfied  70 
Moderately  Satisfied  50 
Fairly  Dissatisfied  30 
Extremely  Dissatisfied  10 
Lowest  Satisfaction  Possible  0 


With  the  criteria  and  scale  established  for  the 
application  of  the  model  the  scores  from  a  user  satisfaction 
survey  can  be  put  to  use  in  determining  a  ranking  of  the 
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various  alternatives.  Prior  to  ranking  alternatives  the 
concordance  and  discordance  thresholds  must  be  selected. 

Here  the  values  of  P=.6  and  Q=.3  are  used  for  the  respective 
thresholds.  For  this  example  the  scores  for  the  different 
alternatives,  given  by  criteria,  are  displayed  in  Table  6. 


Table  6.  Sample  Criteria  Score 


Alternative 

Criteria 

1 

2 

3 

Understandabi 1 i ty 

90 

70 

30 

Ease  of  learning 

70 

70 

50 

Key  mapping 

70 

50 

10 

Availability 

30 

50 

50 

Response  time 

50 

30 

50 

b.  Ranking  Alternatives 

From  the  example  C(l,2)  =  (.2  +.1  +.3)/l  =  .6 
and  D(l,2)  =  20/100  =  .2,  therefor  C(l,2)  =  P  and  D(l,2)  <  Q 
so  that  alternative  one  outranks  alternative  two.  Likewise, 
C(I,3)  =  P  and  D(l,3)  <  Q  so  that  alternative  one  also 
outranks  alternative  three.  In  the  comparison  of 
alternatives  two  and  three,  C(2,3)  =  P  and  0(2,3)  <  Q  so 
that  alternative  two  also  outranks  alternative  three.  The 
final  ranking  of  alternatives  is  then:  1,  2,  and  3.  The 
concordance  and  discordance  index  figures  for  this  example 
are  given  in  Table  7. 
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Table  7.  Concordance  and  Discordance  Figures 


E.  SUMMARY 

User  satisfaction  analysis  is  based  on  user  satisfaction 
surveys  or  questionnaires  as  illustrated  by  Bailey  and 
Pearson's  questionnaire  on  computer  user  satisfaction.  The 
results  of  such  surveys  are  interpreted  through  the  use  of 
formulas  or  models  such  as  Bailey  and  Pearson's  formula,  the 
Elect re  model,  or  MALV.  Prior  to  the  application  of  these 
formulas  the  responses  to  the  surveys  have  to  be  quantified 
though  the  use  of  numerical  scales  such  as  the  importance 
and  satisfaction  scales  of  the  Bailey  and  Pearson  tool. 
Measurement  tools  which  may  be  applied  objectively,  may  be 
subjective  measures  of  user  satisfaction  if  the  data  the 
tool  is  based  on  is  obtained  through  surveys  of  user 
satisfaction. 

1.  Precautions  of  User  Satisfaction  Analysis  Methods 
Analysts  employing  measure  of  user  satisfaction 
should  be  cautious  for  the  results  may  be  undesirable  for 
the  several  reasons.  First,  the  adjective  pairs  and  the 
scales  used  with  them  have  different  meanings  to  the 
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different  respondents  of  the  same  questionnaire  (Galletta 
and  Lederer,  1989).  Second,  the  overall  score  produced  by 
some  techniques  through  the  summation  of  responses  to 
specific  functions  of  an  information  system  is  invalid 
(Galletta  and  Lederer, 1989) .  The  functions  are  assumed  to 
be  homogeneous  when  in  fact  they  are  heterogeneous  which 
makes  a  score  based  on  their  summation  illogical  (Galletta 
and  Lederer, 1989) .  If  the  factors  overlap,  or  are  not 
independent  of  each  other,  then  a  tally  of  the  various 
functions  simply  does  not  make  sense.  Overall  scores  are 
also  arrived  at  without  questioning  the  users'  overall 
perception  of  the  information  system  which  reinforces  a 
concern  that  measures  of  user  satisfaction  are  unsound 
(Galletta  and  Lederer, 1989) .  Analysts  need  to  avoid  methods 
that  violate  measurement  theory  by  the  manner  in  which  the 
methods  call  for  parametric  statistical  analysis  on  ordinal 
measures  such  as  adjective  pairs  used  in  surveys.  Lastly, 
officials  administering  the  survey  can  sway  the  results  by 
influencing  the  setting  or  condition  in  which  the  survey  is 
given  (Galletta  and  Lederer ,1989) . 

2.  Benefits  of  User  Satisfaction  Analysis 

Using  the  Electre  model  to  rank  alternatives  is  a 
legitimate  use  of  ordinal  measures.  As  such  the  results  of 
the  Electre  model  can  augment  the  findings  of  other  analysis 
methods  in  the  analysis  of  the  non-cost  factors, 
particularly  user  acceptance.  Thus  there  is  a  method  that 
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can  be  used  for  the  analysis  of  user  acceptance  as  required 
by  OSA  for  the  analysis  of  alternatives  in  the  acquisition 
of  federal  ADP  systems. 
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VI.  DISCUSSION  AMO  OOHCLUSIOH 


In  the  proceeding  chapters  different  approaches  to 
benefit  analysis  were  discussed  that  could  be  used  in  the 
analysis  of  the  different  non-cost  factors  that  are  required 
by  the  OSA  to  be  evaluated  during  an  analysis  of 
alternatives  prior  to  the  federal  acquisition  of  an  ADP 
system.  The  two  approaches  reviewed  were  benefit  comparison 
and  user  satisfaction.  Both  of  these  strategies  have  their 
own  advantages,  disadvantages,  and  possible  applications 
within  the  DoD  and  the  federal  government  for  the  evaluation 
of  proposed  information  systems. 

A.  BENEFIT  COMPARISON 

1.  Advantages  of  Benefit  Comparison 

The  benefit  comparison  methods  reviewed  were: 
additive  weight,  eigenvector,  and  multi-attribute  utility 
models,  and  each  have  some  capacity  for  sensitivity 
analysis.  They  also  have  the  advantage  that  they  are 
objective  methods  of  benefit  analysis  as  apposed  to  user 
satisfaction.  As  mentioned  the  multi-attribute  utility 
model  is  normative  as  it  is  based  upon  different  aximns. 

2.  Disadvantages  of  Benefit  Coaiparison 

A  disadvantage  of  the  benefit  comparison  methods  is 
that  additive  weight  and  eigenvector  models  do  not  have  the 
capacity  to  consider  risk  or  uncertainty  when  utilised  with 
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the  attributes  given  in  Table  1.  Yet  even  more  critical  is 
that  these  methods  can  not  be  used  in  all  situations.  As 
addressed,  the  multi-attribute  utility  model  is  incompatible 
with  a  situation  where  the  attributes  are  interdependent. 

All  of  the  benefit  comparison  methods  also  require  that  an 
investment  of  time  and  effort  be  made  to  defining  the 
advantages  to  be  gained  from  the  various  proposed  system  so 
that  those  desired  advantages  can  be  used  as  attributes  in 
the  analysis. 

3.  Feasible  Applications  of  Benefit  Comparison 

Benefit  comparison  techniques  are  most  applicable  in 
situations  where  the  users  are  willing  and  able  to  devote 
the  time  and  expense  of  defining  the  desired  benefits  of  a 
proposed  information  system.  For  a  small  application  that 
can  be  developed  in-house  there  may  not  exist  the  need  for  a 
detailed  benefit  analysis.  Benefit  comparison  techniques 
may  however,  be  quite  suitable  for  the  evaluation  of  large 
information  systems  dealing  with  administrative  activities 
or  logistics  that  are  not  excluded  from  the  Brooks  Act  and 
where  the  OSA  recomnended  non-cost  factors  must  be  evaluated 
in  the  analysis  of  alternatives  for  a  federal  acquisition  of 
an  ADP  system.  The  OSA  functional  and  risk  factors  can  then 
be  used  as  the  attributes  in  the  application  of  any  one  of 
the  benefit  comparison  method  discussed. 
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B.  USER  SATISFACTION 


1.  Advantages  of  User  Satisfaction 

The  main  advantage  of  the  user  satisfaction  is  that 
through  this  technique  it  is  easy  to  collect  data  on  the 
users'  perceptions  of  an  existing  information  system.  This 
same  concept  can  be  applied  to  gaining  information  on 
prototypes  of  proposed  information  systems.  The  ease  of 
data  collection  is  in  reference  to  the  fact  that  it  is 
easier  to  measure  a  respondent's  answers  to  a  questionnaire 
than  finding  and  proving  a  cause  and  effect  relationship 
between  a  system  capability  and  any  specific  organizational 
benefit.  The  advantage  of  easy  data  collection,  used  in 
conjunction  with  such  methods  as  the  Electre  model,  enables 
an  analyst  to  employ  user  satisfaction  to  rank  order  several 
competing  alternatives,  and  to  do  all  of  this  from  a  users 
point  of  view. 

2.  Disadvantages  of  User  Satisfaction 

A  disadvantage  of  user  satisfaction  is  that  the 
technique  has  been  used  incorrectly,  as  in  the  Bailey  and 
Pearson  instrument.  Analyst  should  be  cautioned  when  using 
user  satisfaction  not  to  confuse  ordinal  measures,  such  as 
adjective  pairs,  as  interval  measures,  such  as  time  or 
distance.  Analysts  should  also  question  methods  that  score 
alternatives  through  the  stunnation  of  interdependent 
attributes. 
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other  disadvantages  of  the  measures  of  user 
satisfaction  include  having  the  responses  to  the 
questionnaires  affected  by  the  manner  in  which  the  surveys 
were  administered.  Inaccurate  responses  may  also  be  caused 
by  what  is  seen  to  the  respondent's  as  confusing  wording 
used  in  the  questionnaire.  Another  disadvantage  centers 
around  the  evaluation  of  proposed  system  alternatives  versus 
evaluating  existing  systems.  Of  the  literature  cited  in 
Chapter  V  on  user  satisfaction  all  of  the  surveys  used  were 
intended  for  existing  systems.  While  this  technique  may  be 
selected  by  some  to  evaluate  system  effectiveness,  the  user 
satisfaction  method  of  benefit  analysis  appears  to  be  of 
little  use  in  the  analysis  of  proposed  information  systems 
unless  the  analysis  is  tailored  so  that  it  can  be  conducted 
on  a  prototype  of  the  system. 

3.  Feasible  Applications  of  User  Satisfaction 

One  arena  for  the  application  of  user  satisfaction 
as  a  technique  of  evaluating  proposed  information  systems  is 
to  survey  test  bed  organizations  working  with  prototypes  of 
the  different  alternatives  for  new  proposed  system.  These 
organizations  could  receive  experience  with  the  different 
prototypes  and  would  be  knowledgeable  users  who  could  make 
informed  responses  to  surveys  on  the  competing  alternatives. 
Using  measurements  of  user  satisfaction  in  a  test  bed 
organization  would  be  useful  for  systems  that  have  a  high 
degree  of  user  interface,  such  as  in  decision  support 
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systems  or  avionic  software.  The  members  of  these  type  of 
organisations  can  offer  significant  improvements  to  reduce 
ambiguity  through  their  recommendations  concerning:  screens 
lay  outs,  system  response  messages,  or  on-line  help.  Due  to 
the  large  number  of  units  in  the  military  that  perform 
similar  actions,  this  technique  could  be  used  to  evaluate 
highly  interactive  systems,  from  the  users  perspective, 
prior  to  a  large  scale  purchase.  Again,  avionic  or  some 
weapon  system  software  is  a  good  example  where  the  economies 
of  scale  for  the  military  could  make  this  type  of 
prototyping  really  pay  off.  This  use  of  prototyping  may 
help  preclude  the  purchase  of  inappropriate  or  insufficient 
software. 

C.  CONHECTIRG  BENEFITS  TO  ORGANIZATIONAL  GOALS 

Buss  (Buss, 1983)  recommended  taking  into  consideration 
how  well  an  alternative  is  in  line  with  the  organizational 
objectives.  This  is  a  vital  concern  and  should  be  addressed 
by  analysts  when  conducting  a  benefit  analysis.  By 
initiating  a  benefit  analysis  with  a  review  of 
organizational  goals  an  analyst  should  be  aware  of  the  goals 
that  recommendations  should  address  and  take  into  account. 
The  organization's  goals  can  also  be  used  as  a  guide  to 
narrow  in  on  what  system  attributes  need  to  be  evaluated 
during  the  benefit  analysis.  In  Seidmann  and  Arbel 
(Seidmann  and  Arbel, 1984)  a  process  was  presented  that 
hierarchically  decomposed  the  problem  of  defining  system 
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attributes.  First,  the  overall  organizational  objectives 
were  reviewed  to  determine  what  type  of  systems  are  needed. 
In  this  example  a  business  was  examined  and  it  was 
determined  that  it  was  in  need  of  an  accounting  information 
system.  Second,  the  organizational  needs  were  studied  in 
respect  to  the  proposed  system.  This  step  looked  at  major 
fimctions  of  the  planned  system,  such  as  accounts 
receivable,  payroll,  general  ledger,  and  inventory  control. 
Third,  specific  system  operations  were  addressed  that  were 
required  to  support  the  major  functions  defined  earlier. 

Here  such  operations  were  identified  as  file  updating  and 
report  generation.  Finally,  the  system  attributes  were 
defined  that  were  required  to  support  the  specific  system 
functions  determined  in  the  previous  step.  Some  of  the 
attributes  identified  in  this  step  were:  CPU  capacity,  main 
memory  size,  compatibility  of  software,  and  vendor  support. 
(Seidmann  and  Arbel,1984) 

D.  DIFFERENT  APPROACHES  USED  TOGETHER 

There  are  advantages  to  using  the  different  approaches 
to  benefit  analysis  in  concert  with  each  other  and 
capitalizing  on  the  strong  points  of  each.  Where  system 
capabilities  could  easily  be  measured  in  dollar  values, 
present  value  analysis  could  be  employed  to  take  advantage 
of  that  method's  widely  imderstood  methodology.  Where 
system  benefits  are  difficult  to  quantify  in  dollar  term 
benefit  comparison  methods,  such  as  additive  weight,  could 
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be  used  to  take  advantage  of  their  objective  nature  and 
capability  for  sensitivity  analysis.  The  Electre  model 
could  be  used  to  rank  order  the  alternatives  in  the  analysis 
of  user  acceptance  of  a  prototype  where  not  only  are 
benefits  difficult  to  quantify  in  dollar  term,  but  where 
analysts  need  to  be  cautious  with  their  use  of  ordinal 
measures  in  an  analysis.  By  conducting  an  analysis  that 
includes:  present  value  analysis  of  cost  factors,  using  the 
additive  weight  model  to  analyze  non-cost  factors,  and  using 
the  Electre  model  to  specifically  analyze  user  acceptance,  a 
more  complete  picture  can  be  presented  in  the  analysis  of 
alternatives.  This  will  allow  for  a  more  informed  decision 
in  the  final  selection  of  alternatives.  Both  the  Eletre 
model  and  present  value  analysis  could  be  applied  toward 
this  type  combined  analysis  without  modification.  The 
additive  weight  model  would  only  require  slight  changes  from 
the  example  given  previously. 

E.  ADDITIVE  WEIGHT  MODEL  IK  FEDERAL  ACQUISITIONS 

The  fimctional  and  risk  factor  defined  by  the  OSA  would 
be  used  in  the  application  of  the  additive  weight  model  for 
the  analysis  of  non-cost  factors.  The  functional  factor  of 
user-acceptance  would  be  excluded  from  analysis  \uider  the 
additive  weight  model  and  reserved  for  the  Electre  model  so 
as  to  preclude  the  wrongful  use  of  ordinal  measures.  The 
break  down  of  non-cost  factors  in  an  additive  weight 
analysis  would  resemble  Table  8.  Starting  with  these 
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defined  attributes  and  categories  an  analyst  would  then  have 
to  ask  the  decision  maker  to  assign  weights  between  0  and  1 
to  each  of  the  attributes  and  categories.  The  sum  of  the 
weights  assigned  to  the  two  categories  would  have  to  equal  1 
and  the  sum  of  the  weights  assigned  to  the  attribute  of  each 
category  would  also  have  to  equal  1.  Sample  weights  are 
given  in  Table  8. 


Table  8  .  Sample  Weights  for  Non-Cost  Categories  and 
Attributes 


Category 

Attribute 

Weights 

Z1 

Functional 

Factors 

.60 

XI 

Obsolescence 

.08 

X2 

Availability 

.08 

X3 

Reliability 

.11 

X4 

Maintainability 

.16 

X5 

Expandability 

.11 

X6 

Flexibility 

.10 

X7 

Security 

.16 

X8 

Privacy 

.07 

X9 

Personnel  Impacts 

.06 

XIO 

Account abi 1 i t y 

.07 

Z2 

Risk  Factors 

.40 

Xll 

Financial 

.20 

X12 

Technical 

.50 

X13 

Schedule 

.30 

1.  Sample  Scores  for  the  Additive  Weight  Model 

The  next  step  in  applying  the  additive  weight  model 
is  to  score  the  alternatives  one  attribute  at  a  time.  A 
range  of  0  to  1  is  used  for  the  individual  attribute  scores. 
If  the  first  13  attribute  scores  given  in  Table  2  are  used 
as  an  example,  in  combination  with  the  category  and 
attribute  weights  stated  in  Table  8,  then  the  total  weighted 
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score  for  alternative  A  would  be  0.9272  as  expressed  in 
Table  9. 


Table  9.  Calculation  of  the  Weighted  Score  for 
Alternative  A 


Attribute 

Category  x 
Weight 

Attribute 

Weight 

X  Attribute 
Score 

-  Weighted 
Score 

XI 

0.6 

.08 

0.8 

.0384 

X2 

0.6 

.08 

0.8 

.0384 

X3 

0.6 

.11 

0.8 

.0528 

X4 

0.6 

.16 

1.0 

.0960 

X5 

0.6 

.11 

1.0 

.0660 

X6 

0.6 

.10 

0.8 

.0480 

X7 

0.6 

.16 

1.0 

.0960 

X8 

0.6 

.07 

0.9 

.0378 

X9 

0.6 

.06 

1.0 

.0360 

XIO 

0.6 

.07 

0.9 

.0378 

.5472 

XII 

0.4 

0.2 

0.9 

.0720 

XI 2 

0.4 

0.5 

l.( 

.2000 

XI3  0.4 

Total  Weighted  Score 

0.3 

0.9 

.1080 

.3800 

.9272 

2.  Advantage  of  this  Application 

The  additive  weight's  recognised  weakness,  that  of 
not  taking  into  account  risk  or  uncertainty,  is  counteracted 
in  this  application  of  the  model.  The  weakness  is  cancel 
because  the  three  different  risk  factors;  financial, 
technical,  and  schedule,  are  specifically  addressed  as 
attributes  when  using  the  OSA  recommended  non-cost  factors. 
This  application  of  the  additive  weight  model  has  the 
advantages  of  being:  conceptually  simplistic,  calculable  on 
any  standard  spreadsheet  application,  and  in  accordance  with 
the  guidelines  set  forth  by  the  GSA  for  federal  ADP 
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acquisitions.  The  results  of  this  application  of  the 
additive  weight  model  combined  together  with  the  results  of 
present  value  analysis  and  user  acceptance  analysis,  through 
the  use  of  the  Electre  model,  offer  a  more  complete 
evaluation  of  alternatives  than  any  of  the  methods  could 
offer  when  used  alone. 

P.  CONCLUSION 

One  method  of  benefit  analysis  is  not  suitable  for  all 
situations  where  proposed  information  systems  are  in  need  of 
evaluation.  Nor  is  one  method  alone  appropriate  for  the 
evaluation  of  all  of  the  factors  that  must  be  taking  into 
account  during  an  analysis  of  alternative  systems.  Each 
analysis  method  has  its  own  advantages  and  disadvantages. 
Gains  can  be  made  by  combining  methods  to  take  advantage  of 
the  respective  potentials  of  each.  Using  different  methods 
together  would  also  provide  a  more  complete  analysis  as  some 
methods  used  alone  are  inappropriate  for  certain 
applications,  such  attempting  to  use  present  value  analysis 
in  an  evaluation  that  includes  the  measurement  of  user 
acceptance.  System  analysts  conducting  analysis  of 
alternatives  for  a  federal  acquisition  of  information 
systems  will  have  to  be  knowledgeable  of  the  different 
benefit  analysis  methods.  To  obtain  the  required  approval 
for  development  of  information  systems,  and  to  abide  by  the 
OSA  requirements  for  analysis  of  alternatives,  system 
analysis  will  also  have  to  know  how  to  implement  the  right 
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mix  of  benefit  analysis  methods  so  to  fully  evaluate  both 
cost  and  non-cost  factors. 
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APPENDIX  A:  EIOEMVECTOR  MATRICES 


Figure 


Figure 


2.  Pairwise  Comparison  of  Attributes  by  Categories 
Hardware 


XI 

X2 

X3 

X4 

X5 


Software 


X6 

X7 

XS 


System  Support 

X9  XIO  XII  XI2  XI3  XI 4 


X9 

XIO 

XII 
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3.  Pairwise  Comparison  of  Categories 


ZI 

Z2 

Z3 


Zl  Z2  Z3 
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5 

3 

3 

4 
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1 

3 

2 
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1 
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3 
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APPENDIX  B:  C(»1PUTER  USER  SATISFACTION  QUESTIONNAIRE 


Use  the  following  scale  in  response  to  each  pair  of 
adjectives : 


!  extremely!  quite!  slightly  .'equally!  slightly  {quite  I  extremely! 

1.  Top  management  involvement:  The  positive  or  negative 
degree  of  interest,  enthusiasm,  support,  or  participation  of 
any  management  level  above  the  user's  own  level  toward 
computer-based  information  systems  or  services  or  toward  the 
computer  staff  which  supports  them. 


strong 

consistent 

good 

significant 

satisfactory 

important 


weak 

inconsistent 

bad 

insignificant 

unsatisfactory 

unimportant 


2.  Organizational  competition  with  the  EDP  unit:  The 
contention  between  the  respondent's  organizational  unit  and 
the  EDP  unit  when  vying  for  organizational  resources  or  for 
responsibility  for  success  or  failure  of  computer-based 
information  systems  or  services  of  interest  to  both  parties. 


productive 

rational 

low 

harmonious 

satisfactory 

important 


destructive 

emotional 

high 

dissonant 

unsatisfactory 

unimportant 


3.  Priorities  determination:  Policies  and  procedures  which 
establish  precedence  for  the  allocation  of  EDP  resources  and 
services  between  different  organizational  units  and  their 
requests . 


fair 

consistent 

just 

precise 

satisfactory 

important 


unfair 

inconsistent 

unjust 

vague 

unsatisfactory 

unimportant 
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4.  Charge-back  method  of  payment  for  services:  The 
schedule  of  charges  and  the  procedures  for  assessing  users 
on  a  pro  rata  basis  for  the  EDP  resources  and  services  that 
they  utilize. 


just 

reasonable 

consistent 

known 

satisfactory 

important 


unjust 

unreasonable 

inconsistent 

unknown 

unsatisfactory 

unimportant 


5.  Relationship  with  the  EDP  staff:  The  manner  and  methods 
of  interaction,  conduct,  and  association  between  the  user 
and  the  EDP  staff. 


harmonious 

good 

cooperative 

candid 

satisfactory 

important 


dissonant 

bad 

uncooperative 

deceitful 

unsatisfactory 

unimportant 


6.  Comnuni cation  with  the  EDP  staff:  The  manner  and 
methods  of  information  exchange  between  the  user  and  the  EDP 
staff . 


harmonious 

productive 

precise 

meaningful 

satisfactory 

important 


dissonant 

destructive 

vague 

meaningless 

unsatisfactory 

unimportant 


7.  Technical  competence  of  the  EDP  staff:  The  computer 
technology  skills  and  expertise  exhibited  by  the  EDP  staff. 


current 

sufficient 

superior 

high 

satisfactory 

important 


obsolete 

insufficient 

inferior 

low 

unsatisfactory 

xinimportant 


74 


8.  Attitude  of  the  EDP  staff:  The  willingness  and 
convnitment  of  the  EDP  staff  to  subjugate  external, 
professional  goals  in  favor  of  organizationally  directed 
goals  and  tasks. 


user-oriented 

cooperative 

courteous 

positive 

satisfact'^i.y 

important 


self-centered 

belligerent 

discourteous 

negative 

unsatisfactory 

unimportant 


9.  Schedule  of  products  and  services:  The  EDP  center 
timetable  for  production  of  information  system  outputs  and 
for  provision  of  computer-based  services. 


good 
regul ar 
reasonable 
acceptable 
satisfactory 
important 


bad 

irregular 

unreasonable 

unacceptable 

unsatisfactory 

unimportant 


10.  Time  required  for  new  development;  The  elapsed  time 
between  the  user's  request  for  new  applications  and  the 
design,  development,  and/or  implementation  of  the 
application  systems  by  the  EDP  staff. 


short 
dependabl e 
reasonable 
acceptable 
satisfactory 
important 


long 

undependable 

unreasonable 

unacceptable 

unsatisfactory 

unimportant 


11.  Processing  of  change  requests:  The  manner,  method,  and 
required  time  with  which  the  EDP  staff  responds  to  user 
requests  for  changes  in  existing  computer-based  information 
systems  or  services. 


fast 

timely 

simple 

flexible 

satisfactory 

important 


slow 

untimely 

complex 

rigid 

unsatisfactory 

unimportant 
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12.  Vendor  support:  The  type  and  quality  of  the  service 
rendered  by  a  vendor,  either  directly  or  indirectly,  to  the 
user  to  maintain  the  hardware  or  software  required  by  that 
organizational  status. 


skilled 

sufficient 

eager 

consistent 

satisfactory 

important 


bungling 

insufficient 

indifferent 

inconsistent 

unsatisfactory 

unimportant 


13.  Response/ turnaround  time:  The  elapsed  time  between  a 
user-initiated  request  for  service  or  action  and  a  reply  to 
that  request.  Response  time  generally  refers  to  the  elapsed 
time  for  terminal  type  request  or  entry.  Turnaround  time 
generally  refers  to  the  elapsed  time  for  execution  of  a 
program  submitted  or  requested  by  a  user  and  the  return  of 
the  output  to  that  user. 


fast 

good 

consistent 

reasonable 

satisfactory 

important 


slow 

bad 

inconsistent 

unreasonable 

unsatisfactory 

unimportant 


14.  Means  of  input/output  with  EDP  center:  The  method  and 
medium  by  which  a  user  inputs  data  to  and  receives  output 
from  the  EDP  center. 


convenient 

clear 

efficient 

organized 

satisfactory 

important 


inconvenient 

hazy 

inefficient 

disorganized 

unsatisfactory 

unimportant 


15.  Convenience  of  access:  The  ease  of  difficulty  with 
which  the  user  may  act  to  utilize  the  capability  of  the 
computer  system. 


convenient 

good 

easy 

efficient 

satisfactory 

important 


inconvenient 

bad 

dif f icul t 
inefficient 
unsatisfactory 
unimportant 
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16.  Accuracy:  The  correctness  of  the  output  information. 


accurate 

high 

consistent 

sufficient 

satisfactory 

important 


inaccurate 

low 

inconsistent 

insufficient 

unsatisfactory 

unimportant 


17.  Timeliness:  The  availability  of  the  output  information 
at  a  time  suitable  for  its  use. 


timely 
reasonabl e 
consistent 
punctual 
satisfactory 
important 


untimely 

unreasonable 

inconsistent 

tardy 

unsatisfactory 

unimportant 


18.  Precision:  The  variability  of  the  output  information 
from  that  which  it  purports  to  measure. 


sufficient 

consistent 

high 

definite 

satisfactory 

important 


insufficient 

inconsistent 

low 

uncertain 

unsatisfactory 

unimportrnt 


19.  Reliability:  The  consistency  and  dependability  of  the 
output  information. 


consistent 

high 

superior 

sufficient 

satisfactory 

important 


inconsistent 

low 

inferior 

insufficient 

unsatisfactory 

unimportant 


20.  Currency:  The  age  of  the  output  information. 


good 

timely 

adequate 

reasonable 

satisfactory 

important 


bad 

untimely 

inadequate 

unreasonable 

unsatisfactory 

unimportant 
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21.  Completeness:  The  comprehensiveness  of  the  output 
information  content. 


complete 

consistent 

sufficient 

adequate 

satisfactory 

important 


incomplete 

inconsistent 

insufficient 

inadequate 

unsatisfactory 

unimportant 


22.  Format  of  output:  The  material  design  of  the  layout 
and  display  of  the  output  contents. 


good 

simple 

readable 

useful 

satisfactory 

important 


bad 

complex 

unreadable 

useless 

unsatisfactory 

unimportant 


23.  Language:  The  set  of  vocabulary,  syntax,  and 
grammatical  rules  used  to  interact  with  the  computer 
systems . 


simple 

powerful 

easy 

easy-to-use 

satisfactory 

important 


complex 

weak 

difficult 

hard-to-use 

unsatisfactory 

unimportant 


24.  Volume  of  output:  The  amount  of  information  conveyed 

to  a  user  from  computer-based  systems.  This  is  expressed 
not  only  by  the  number  of  reports  or  outputs  but  also  by  the 
voluminousness  of  the  output  contents. 


concise 

sufficient 

necessary 

reasonable 

satisfactory 

important 


redundant 

insufficient 

unnecessary 

unreasonable 

xinsatisf  actory 

unimportant 
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25.  Relevancy:  The  degree  of  congruence  between  what  the 
user  wants  or  requires  and  what  is  provided  by  the 
information  products  services. 


useful 

relevant 

clear 

good 

satisfactory 

important 


useless 

irrelevant 

hazy 

bad 

unsatisfactory 

unimportant 


26.  Error  recovery:  The  methods  and  policies  governing 
correction  and  return  of  system  outputs  that  are  incorrect. 


fast 

superior 

complete 

simple 

satisfactory 

important 


slow 

inferior 

incomplete 

complex 

unsatisfactory 

unimportant 


27.  Security  of  data:  The  safeguarding  of  data  from 
misappropriation  or  unauthorized  alteration  or  loss. 


secure 

good 

definite 

complete 

satisfactory 

important 


insecure 

bad 

uncertain 

incomplete 

unsatisfactory 

unimportant 


28.  Documentation:  The  recorded  description  of  an 
information  system.  This  includes  formal  instructions  for 
the  utilization  of  the  system. 


clear 

available 

complete 

current 

satisfactory 

important 


hazy 

unavailable 

incomplete 

obsolete 

unsatisfactory 

unimportant 
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29.  Expectations:  The  set  of  attributes  or  features  of  the 
computer-based  information  products  or  services  that  a  user 
considers  reasonable  and  due  from  the  computer-based 
information  support  rendered  within  the  organization. 


pleased 

high 

definite 

optimistic 

satisfactory 

important 


displeased 

low 

uncertain 

pessimistic 

unsatisfactory 

unimportant 


30.  Understanding  of  systems:  The  degree  of  comprehension 
that  a  user  possesses  about  the  computer-based  information 
system  or  services  that  are  provided. 


high 

sufficient 

complete 

easy 

satisfactory 

important 


low 

insufficient 

incomplete 

hard 

unsatisfactory 

unimportant 


31.  Perceived  utility:  The  user's  judgement  about  the 
relative  balance  between  the  cost  and  the  considered 
usefulness  of  the  computer-based  information  products  or 
services  that  are  provided.  The  costs  include  any  costs 
related  to  providing  the  resource,  including  money,  time, 
manpower,  and  opportunity.  The  usefulness  includes  any 
benefits  that  the  user  believes  to  be  derived  from  the 
support . 


high 

positive 

sufficient 

useful 

satisfactory 

important 


low 

negative 

insufficient 

useless 

unsatisfactory 

unimportant 


32.  Confidence  in  the  system:  The  user's  feelings  of 
assurance  or  certainty  about  the  systems  provided. 


high 

strong 

definite 

good 

satisfactory 

important 


low 

weak 

uncertain 

bad 

unsatisfactory 

\inimportant 
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33.  Feeling  of  participation:  The  degree  of  involvement 
and  commitment  which  the  user  shares  with  the  EDP  staff  and 
others  toward  the  functioning  of  the  computer-based 
information  systems  and  services. 


positive  j. 
encouraged  J. 
sufficient  j. 
involved  1 
satisfactory  1 
important  JL 


negative 

repelled 

insufficient 

uninvolved 

unsatisfactory 

unimportant 


34.  Feeling  of  control:  The  user's  awareness  of  the 
personal  power  or  lack  of  power  to  regulate,  direct  or 
dominate  the  development,  alteration,  and/or  execution  of 
the  computer-based  information  systems  or  services  which 
serve  the  user's  perceived  function. 


high  1 
sufficient  1 
precise  1 
strong  1 
satisfactory  1 
important  1 


low 

insufficient 

vague 

weak 

unsatisfactory 

unimportant 


35.  Degree  of  training:  The  amount  of  specialized 
instruction  and  practice  that  is  afforded  to  the  user  to 
increase  the  user's  proficiency  in  utilizing  the  computer 
capability  that  xs  unavailable. 


complete 

sufficient 

high 

superior 

satisfactory 

important 


incomplete 

insufficient 

low 

inferior 

unsatisfactory 

unimportant 


36.  Job  effects:  The  changes  in  job  freedom  and  job 
performance  that  are  ascertained  by  the  user  as  resulting 
from  modifications  induced  by  the  computer-based  information 
system  and  services. 


1 iberating 
significant 
good 
valuable 
satisfactory 
important 


inhibiting 

insignificant 

bad 

worthless 

unsatisfactory 

unimportant 
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37 .  Organizational  position  of  the  EDP  function:  The 
hierarchical  relationship  of  the  EDP  function  to  the  overall 
organizational  structure. 


appropriate 

strong 

clear 

progressive 

satisfactory 

important 


inappropriate 

weak 

hazy 

regressive 

unsatisfactory 

unimportant 


38.  Flexibility  of  systems:  The  capacity  of  the 
information  system  to  change  or  to  adjust  in  response  to  new 
conditions,  demands,  or  circumstances. 


flexible 

versatile 

sufficient 

high 

satisfactory 

important 


rigid 

limited 

insufficient 

low 

unsatisfactory 

unimportant 


39.  Integration  of  systems:  The  ability  of  systems  to 
communicate/ transmit  data  between  systems  servicing 
different  functional  areas. 


complete 

sufficient 

successful 

good 

satisfactory 

important 


incomplete 

insufficient 

unsuccessful 

bad 

unsatisfactory 

unimportant 


The  above  user  satisfaction  questionnaire  is  basis  of  a 
copyrighted  measurement  instrument  and  is  reprinted  here 
only  as  an  educational  example.  Application  of  the 
measurement  instrument,  other  than  in  research,  should  be 
preceded  by  the  receipt  of  permission  from  Dr.  Pearson. 
(Bailey  and  Pearson, 1983) 
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